Re: [DISCUSS] Release Flink 1.16.3

[Konstantin Knauf]

+1. Thank you for picking it up. Yes, this will be the final bug fix for
Flink 1.16.

Am Mo., 6. Nov. 2023 um 03:47 Uhr schrieb Rui Fan <1996fan...@gmail.com>:

> Hi all,
>
> I would like to discuss creating a new 1.16 patch release (1.16.3). The
> last 1.16 release is over five months old, and since then, 50 tickets have
> been closed [1], of which 10 are blocker/critical [2]. Some
> of them are quite important, such as FLINK-32296 [3], FLINK-32548 [4]
> and FLINK-33010[5].
>
> In addition to this, FLINK-33149 [6] is important to bump snappy-java to
> 1.1.10.4.
> Although FLINK-33149 is unresolved, it was done in 1.16.3.
>
> I am not aware of any unresolved blockers and there are no in-progress
> tickets [7]. Please let me know if there are any issues you'd like to be
> included in this release but still not merged.
>
> Since 1.18.0 has been released, I'd suggest that we vote to make 1.16.3
> the final bugix release of 1.16, looking forward to any feedback from you.
> Background info could be found at [8], and thanks Jing for the information.
>
> If the community agrees to create this new patch release, I could
> volunteer as the release manager.
>
> Since there will be another flink-1.17.2 release request during the same
> time,
> I will work with Yun and Yu since many issues will be fixed in both
> releases.
>
> [1]
>
> https://issues.apache.org/jira/browse/FLINK-32231?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.16.3%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC
> [2]
>
> https://issues.apache.org/jira/browse/FLINK-32231?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.16.3%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC
> [3] https://issues.apache.org/jira/browse/FLINK-32296
> [4] https://issues.apache.org/jira/browse/FLINK-32548
> [5] https://issues.apache.org/jira/browse/FLINK-33010
> [6] https://issues.apache.org/jira/browse/FLINK-33149
> [7] https://issues.apache.org/jira/projects/FLINK/versions/12353259
> [8] https://lists.apache.org/thread/szq23kr3rlkm80rw7k9n95js5vqpsnbv
>
> Best,
> Rui
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Release 1.17.2

[Konstantin Knauf]

Thank you for picking it up! +1

Cheers,

Konstantin

Am Mo., 6. Nov. 2023 um 03:48 Uhr schrieb Yun Tang :

> Hi all,
>
> I would like to discuss creating a new 1.17 patch release (1.17.2). The
> last 1.17 release is near half a year old, and since then, 79 tickets have
> been closed [1], of which 15 are blocker/critical [2]. Some
> of them are quite important, such as FLINK-32758 [3], FLINK-32296 [4],
> FLINK-32548 [5]
> and FLINK-33010[6].
>
> In addition to this, FLINK-33149 [7] is important to bump snappy-java to
> 1.1.10.4.
> Although FLINK-33149 is unresolved, it was done in 1.17.2.
>
> I am not aware of any unresolved blockers and there are no in-progress
> tickets [8]. Please let me know if there are any issues you'd like to be
> included in this release but still not merged.
>
> If the community agrees to create this new patch release, I could
> volunteer as the release manager with Yu Chen.
>
> Since there will be another flink-1.16.3 release request during the same
> time, we will work with Rui Fan since many issues will be fixed in both
> releases.
>
> [1]
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC
> [2]
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC
> [3] https://issues.apache.org/jira/browse/FLINK-32758
> [4] https://issues.apache.org/jira/browse/FLINK-32296
> [5] https://issues.apache.org/jira/browse/FLINK-32548
> [6] https://issues.apache.org/jira/browse/FLINK-33010
> [7] https://issues.apache.org/jira/browse/FLINK-33149
> [8] https://issues.apache.org/jira/projects/FLINK/versions/12353260
>
> Best
> Yun Tang
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-364: Improve the restart-strategy

[Konstantin Knauf]

Hi Rui,

Thank you for this proposal and working on this. I also agree that
exponential back off makes sense as a new default in general. I think
restarting indefinitely (no max attempts) makes sense by default, though,
but of course allowing users to change is valuable.

So, overall +1.

Cheers,

Konstantin

Am Di., 17. Okt. 2023 um 07:11 Uhr schrieb Rui Fan <1996fan...@gmail.com>:

> Hi all,
>
> I would like to start a discussion on FLIP-364: Improve the
> restart-strategy[1]
>
> As we know, the restart-strategy is critical for flink jobs, it mainly
> has two functions:
> 1. When an exception occurs in the flink job, quickly restart the job
> so that the job can return to the running state.
> 2. When a job cannot be recovered after frequent restarts within
> a certain period of time, Flink will not retry but will fail the job.
>
> The current restart-strategy support for function 2 has some issues:
> 1. The exponential-delay doesn't have the max attempts mechanism,
> it means that flink will restart indefinitely even if it fails frequently.
> 2. For multi-region streaming jobs and all batch jobs, the failure of
> each region will increase the total number of job failures by +1,
> even if these failures occur at the same time. If the number of
> failures increases too quickly, it will be difficult to set a reasonable
> number of retries.
> If the maximum number of failures is set too low, the job can easily
> reach the retry limit, causing the job to fail. If set too high, some jobs
> will never fail.
>
> In addition, when the above two problems are solved, we can also
> discuss whether exponential-delay can replace fixed-delay as the
> default restart-strategy. In theory, exponential-delay is smarter and
> friendlier than fixed-delay.
>
> I also thank Zhu Zhu for his suggestions on the option name in
> FLINK-32895[2] in advance.
>
> Looking forward to and welcome everyone's feedback and suggestions, thank
> you.
>
> [1] https://cwiki.apache.org/confluence/x/uJqzDw
> [2] https://issues.apache.org/jira/browse/FLINK-32895
>
> Best,
> Rui
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [ANNOUNCE] The Flink Speed Center and benchmark daily run are back online

[Konstantin Knauf]

Thanks a lot for working on this!

Am Do., 19. Okt. 2023 um 10:24 Uhr schrieb Zakelly Lan <
zakelly@gmail.com>:

> Hi everyone,
>
> Flink benchmarks [1] generate daily performance reports in the Apache
> Flink slack channel (#flink-dev-benchmarks) to detect performance
> regression [2]. Those benchmarks previously were running on several
> machines donated and maintained by Ververica. Unfortunately, those
> machines were gone due to account issues [3] and the benchmarks daily
> run stopped since August 24th delaying the release of Flink 1.18 a
> bit. [4].
>
> Ververica donated several new machines! After several weeks of work, I
> have successfully re-established the codespeed panel and benchmark
> daily run pipelines on them. At this time, we are pleased to announce
> that the Flink Speed Center and benchmark pipelines are back online.
> These new machines have a more formal management to ensure that
> previous accidents will not occur in the future.
>
> What's more, I successfully recovered historical data backed up by
> Yanfei Lei [5]. So with the old domain [6] redirected to the new
> machines, the old links that existed in previous records will still be
> valid. Besides the benchmarks with Java8 and Java11, I also added a
> pipeline for Java17 running daily.
>
> How to use it:
> We also registered a new domain name 'flink-speed.xyz' for the Flink
> Speed Center [7]. It is recommended to use the new domain in the
> future. Currently, the self-service method of triggering benchmarks is
> unavailable considering the lack of resources and potential
> vulnerabilities of Jenkins. Please contact one of Apache Flink PMCs to
> submit a benchmark. More info is updated on the wiki[8].
>
> Daily Monitoring:
> The performance daily monitoring on the Apache Flink slack channel [2]
> is still unavailable as the benchmark results need more time to
> stabilize in the new environment. Once the baseline results become
> available for regression detection, I will enable the daily
> monitoring.
>
> Please feel free to reach out to me if you have any suggestions or
> questions. Thanks Ververica again for denoting machines!
>
>
> Best,
> Zakelly
>
> [1] https://github.com/apache/flink-benchmarks
> [2] https://lists.apache.org/thread/zok62sx4m50c79htfp18ymq5vmtgbgxj
> [3] https://issues.apache.org/jira/browse/FLINK-33052
> [4] https://lists.apache.org//thread/5x28rp3zct4p603hm4zdwx6kfr101w38
> [5] https://issues.apache.org/jira/browse/FLINK-30890
> [6] http://codespeed.dak8s.net:8000
> [7] http://flink-speed.xyz
> [8]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [ANNOUNCE] Release 1.18.0, release candidate #1

[Konstantin Knauf]

Hi everyone,

I've just opened a PR for the release announcement [1] and I am looking
forward to reviews and feedback.

Cheers,

Konstantin

[1] https://github.com/apache/flink-web/pull/680

Am Fr., 6. Okt. 2023 um 11:03 Uhr schrieb Sergey Nuyanzin <
snuyan...@gmail.com>:

> sorry for not mentioning it in previous mail
>
> based on the reason above I'm
> -1 (non-binding)
>
> also there is one more issue [1]
> which blocks all the externalised connectors testing against the most
> recent commits in
> to corresponding branches
> [1] https://issues.apache.org/jira/browse/FLINK-33175
>
>
> On Thu, Oct 5, 2023 at 11:19 PM Sergey Nuyanzin 
> wrote:
>
> > Thanks for creating RC1
> >
> > * Downloaded artifacts
> > * Built from sources
> > * Verified checksums and gpg signatures
> > * Verified versions in pom files
> > * Checked NOTICE, LICENSE files
> >
> > The strange thing I faced is
> > CheckpointAfterAllTasksFinishedITCase.testRestoreAfterSomeTasksFinished
> > fails on AZP [1]
> >
> > which looks like it is related to [2], [3] fixed  in 1.18.0 (not 100%
> > sure).
> >
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-33186
> > [2] https://issues.apache.org/jira/browse/FLINK-32996
> > [3] https://issues.apache.org/jira/browse/FLINK-32907
> >
> > On Tue, Oct 3, 2023 at 2:53 PM Ferenc Csaky 
> > wrote:
> >
> >> Thanks everyone for the efforts!
> >>
> >> Checked the following:
> >>
> >> - Downloaded artifacts
> >> - Built Flink from source
> >> - Verified checksums/signatures
> >> - Verified NOTICE, LICENSE files
> >> - Deployed dummy SELECT job via SQL gateway on standalone cluster,
> things
> >> seemed fine according to the log files
> >>
> >> +1 (non-binding)
> >>
> >> Best,
> >> Ferenc
> >>
> >>
> >> --- Original Message ---
> >> On Friday, September 29th, 2023 at 22:12, Gabor Somogyi <
> >> gabor.g.somo...@gmail.com> wrote:
> >>
> >>
> >> >
> >> >
> >> > Thanks for the efforts!
> >> >
> >> > +1 (non-binding)
> >> >
> >> > * Verified versions in the poms
> >> > * Built from source
> >> > * Verified checksums and signatures
> >> > * Started basic workloads with kubernetes operator
> >> > * Verified NOTICE and LICENSE files
> >> >
> >> > G
> >> >
> >> > On Fri, Sep 29, 2023, 18:16 Matthias Pohl
> matthias.p...@aiven.io.invalid
> >> >
> >> > wrote:
> >> >
> >> > > Thanks for creating RC1. I did the following checks:
> >> > >
> >> > > * Downloaded artifacts
> >> > > * Built Flink from sources
> >> > > * Verified SHA512 checksums GPG signatures
> >> > > * Compared checkout with provided sources
> >> > > * Verified pom file versions
> >> > > * Went over NOTICE file/pom files changes without finding anything
> >> > > suspicious
> >> > > * Deployed standalone session cluster and ran WordCount example in
> >> batch
> >> > > and streaming: Nothing suspicious in log files found
> >> > >
> >> > > +1 (binding)
> >> > >
> >> > > On Fri, Sep 29, 2023 at 10:34 AM Etienne Chauchot
> >> echauc...@apache.org
> >> > > wrote:
> >> > >
> >> > > > Hi all,
> >> > > >
> >> > > > Thanks to the team for this RC.
> >> > > >
> >> > > > I did a quick check of this RC against user pipelines (1) coded
> with
> >> > > > DataSet (even if deprecated and soon removed), DataStream and SQL
> >> APIs
> >> > > >
> >> > > > based on the small scope of this test, LGTM
> >> > > >
> >> > > > +1 (non-binding)
> >> > > >
> >> > > > [1] https://github.com/echauchot/tpcds-benchmark-flink
> >> > > >
> >> > > > Best
> >> > > > Etienne
> >> > > >
> >> > > > Le 28/09/2023 à 19:35, Jing Ge a écrit :
> >> > > >
> >> > > > > Hi everyone,
> >> > > > >
> >> > > > > The RC1 for Apache Flink 1.18.0 has been created. The related
> >> voting
> >> > > > > process will be triggered once the announcement is ready. The
> RC1
> >> has
> >> > > > > all
> >> > > > > the artifacts that we would typically have for a release, except
> >> for
> >> > > > > the
> >> > > > > release note and the website pull request for the release
> >> announcement.
> >> > > > >
> >> > > > > The following contents are available for your review:
> >> > > > >
> >> > > > > - Confirmation of no benchmarks regression at the thread[1].
> >> > > > > - The preview source release and binary convenience releases
> [2],
> >> which
> >> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6
> [3].
> >> > > > > - all artifacts that would normally be deployed to the Maven
> >> > > > > Central Repository [4].
> >> > > > > - source code tag "release-1.18.0-rc1" [5]
> >> > > > >
> >> > > > > Your help testing the release will be greatly appreciated! And
> >> we'll
> >> > > > > create the rc1 release and the voting thread as soon as all the
> >> efforts
> >> > > > > are
> >> > > > > finished.
> >> > > > >
> >> > > > > [1]
> >> https://lists.apache.org/thread/yxyphglwwvq57wcqlfrnk3qo9t3sr2ro
> >> > > > > [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc1/
> >> > > > > [3]https://dist.apache.org/repos/dist/release/flink/KEYS
> >> > > > > [4]
> >> > > > >
> >> 

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

[Konstantin Knauf]

Hi Bonnie,

this makes sense to me, in particular, given that we already have this
toggle for a different type of hints.

Best,

Konstantin

Am Mi., 16. Aug. 2023 um 19:38 Uhr schrieb Bonnie Arogyam Varghese
:

> Hi Liu,
>  Options hints could be a security concern since users can override
> settings. However, query hints specifically could affect performance.
> Since we have a config to disable Options hint, I'm suggesting we also have
> a config to disable Query hints.
>
> On Wed, Aug 16, 2023 at 9:41 AM liu ron  wrote:
>
> > Hi,
> >
> > Thanks for driving this proposal.
> >
> > Can you explain why you would need to disable query hints because of
> > security issues? I don't really understand why query hints affects
> > security.
> >
> > Best,
> > Ron
> >
> > Bonnie Arogyam Varghese  于2023年8月16日周三
> > 23:59写道：
> >
> > > Platform providers may want to disable hints completely for security
> > > reasons.
> > >
> > > Currently, there is a configuration to disable OPTIONS hint -
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-dynamic-table-options-enabled
> > >
> > > However, there is no configuration available to disable QUERY hints -
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/hints/#query-hints
> > >
> > > The proposal is to add a new configuration:
> > >
> > > Name: table.query-options.enabled
> > > Description: Enable or disable the QUERY hint, if disabled, an
> > > exception would be thrown if any QUERY hints are specified
> > > Note: The default value will be set to true.
> > >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl

[Konstantin Knauf]

Congrats, Matthias!

Am Fr., 4. Aug. 2023 um 09:15 Uhr schrieb Paul Lam :

> Congratulation, Matthias!
>
> Best,
> Paul Lam
>
> > 2023年8月4日 15:09，yuxia  写道：
> >
> > Congratulation, Matthias!
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "Yun Tang" 
> > 收件人: "dev" 
> > 发送时间: 星期五, 2023年 8 月 04日 下午 3:04:52
> > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl
> >
> > Congratulation, Matthias!
> >
> >
> > Best
> > Yun Tang
> > 
> > From: Jark Wu 
> > Sent: Friday, August 4, 2023 15:00
> > To: dev@flink.apache.org 
> > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl
> >
> > Congratulations, Matthias!
> >
> > Best,
> > Jark
> >
> > On Fri, 4 Aug 2023 at 14:59, Weihua Hu  wrote:
> >
> >> Congratulations,  Matthias!
> >>
> >> Best,
> >> Weihua
> >>
> >>
> >> On Fri, Aug 4, 2023 at 2:49 PM Yuxin Tan 
> wrote:
> >>
> >>> Congratulations, Matthias!
> >>>
> >>> Best,
> >>> Yuxin
> >>>
> >>>
> >>> Sergey Nuyanzin  于2023年8月4日周五 14:21写道：
> >>>
>  Congratulations, Matthias!
>  Well deserved!
> 
>  On Fri, Aug 4, 2023 at 7:59 AM liu ron  wrote:
> 
> > Congrats, Matthias!
> >
> > Best,
> > Ron
> >
> > Shammon FY  于2023年8月4日周五 13:24写道：
> >
> >> Congratulations, Matthias!
> >>
> >> On Fri, Aug 4, 2023 at 1:13 PM Samrat Deb 
>  wrote:
> >>
> >>> Congrats, Matthias!
> >>>
> >>>
> >>> On Fri, 4 Aug 2023 at 10:13 AM, Benchao Li  >>>
> > wrote:
> >>>
>  Congratulations, Matthias!
> 
>  Jing Ge  于2023年8月4日周五 12:35写道：
> 
> > Congrats! Matthias!
> >
> > Best regards,
> > Jing
> >
> > On Fri, Aug 4, 2023 at 12:09 PM Yangze Guo <
> >> karma...@gmail.com
> 
> >> wrote:
> >
> >> Congrats, Matthias!
> >>
> >> Best,
> >> Yangze Guo
> >>
> >> On Fri, Aug 4, 2023 at 11:44 AM Qingsheng Ren <
>  re...@apache.org>
>  wrote:
> >>>
> >>> Congratulations, Matthias! This is absolutely well
> >>> deserved.
> >>>
> >>> Best,
> >>> Qingsheng
> >>>
> >>> On Fri, Aug 4, 2023 at 11:31 AM Rui Fan <
>  1996fan...@gmail.com>
>  wrote:
> >>>
>  Congratulations Matthias, well deserved!
> 
>  Best,
>  Rui Fan
> 
>  On Fri, Aug 4, 2023 at 11:30 AM Leonard Xu <
> > xbjt...@gmail.com>
> > wrote:
> 
> > Congratulations,  Matthias.
> >
> > Well deserved ^_^
> >
> > Best,
> > Leonard
> >
> >
> >> On Aug 4, 2023, at 11:18 AM, Xintong Song <
>  tonysong...@gmail.com
> >>
>  wrote:
> >>
> >> Hi everyone,
> >>
> >> On behalf of the PMC, I'm very happy to announce
> >> that
> >>> Matthias
> >> Pohl has
> >> joined the Flink PMC!
> >>
> >> Matthias has been consistently contributing to the
> > project
>  since
> >> Sep
> > 2020,
> >> and became a committer in Dec 2021. He mainly works
> >>> in
> >>> Flink's
> > distributed
> >> coordination and high availability areas. He has
> >>> worked
> > on
> >>> many
> >> FLIPs
> >> including FLIP195/270/285. He helped a lot with the
> > release
> >> management,
> >> being one of the Flink 1.17 release managers and
> >> also
> > very
>  active
> >> in
> > Flink
> >> 1.18 / 2.0 efforts. He also contributed a lot to
> > improving
> >>> the
> >> build
> >> stability.
> >>
> >> Please join me in congratulating Matthias!
> >>
> >> Best,
> >>
> >> Xintong (on behalf of the Apache Flink PMC)
> >
> >
> 
> >>
> >
> 
> 
>  --
> 
>  Best,
>  Benchao Li
> 
> >>>
> >>
> >
> 
> 
>  --
>  Best regards,
>  Sergey
> 
> >>>
> >>
>
>

-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

[Konstantin Knauf]

Hi Jing,

> How could we help users and avoid this happening?

I don't think we will be able to avoid this in all cases. And I think
that's ok. Its always a trade-off between supporting new use cases and
moving the project forward and backwards compatibility (in a broad sense).
For example, we dropped Mesos support in a minor release in the past. If
you're only option for running Flink was Mesos, you were stuck on Flink
1.13 or so.

So, I think, it is in the end a case-by-case decision. How big is the cost
of continued support a "legacy feature/system" and how many users are
affected to which degree by dropping it?

Best,

Konstantin


Am Di., 25. Juli 2023 um 18:34 Uhr schrieb Jing Ge
:

> Hi Konstantin,
>
> I might have not made myself clear enough, apologies. The
> source-/sink-function was used as a concrete example to discuss the pattern
> before we decided to offer LTS. The intention was not to hijack this thread
> to discuss how to deprecate them.
>
> We all wish that the only thing users need to migrate from Flink 1.x to 2.0
> is some code changes in their repos and we all wish users will migrate, if
> LTS has long enough support time. But the question I tried to discuss is
> not the wish but the "How?". We might be able to toss the high migration
> effort aside(we shouldn't), since it is theoretically still doable if users
> have long enough time, even if the effort is extremely high. Another
> concern is that if "function regressions" is allowed in 2.0, i.e. if 2.0
> has a lack of functionalities or bugs compared to 1.x, there will be no way
> for users to do the migration regardless of whether we encourage them to
> migrate or they haven been given enough time(how long is enough?) because
> LTS has been offered. How could we help users and avoid this happening?
>
> Best regards,
> Jing
>
> On Tue, Jul 25, 2023 at 6:57 PM Konstantin Knauf 
> wrote:
>
> > Hi Jing,
> >
> > let's not overindex on the Source-/SinkFunction discussion in this
> thread.
> >
> > We will generally drop/break a lot of APIs in Flink 2.0. So, naturally
> > users will need to make more changes to their code in order to migrate
> from
> > 1.x to Flink 2.0. In order to give them more time to do this, we support
> > the last Flink 1.x release for a longer time with bug fix releases.
> >
> > Of course, we still encourage users to migrate to Flink 2.0, because at
> > some point, we will stop support Flink 1.x. For example, if we followed
> > Marton's proposal we would support Flink 1.x LTS for about 2 years
> (roughly
> > 4 minor release cycles) instead of about 1 year (2 minor release cycles)
> > for regular minor releases. This seems like a reasonable timeframe to me.
> > It also gives us more time to discover and address blockers in migrating
> to
> > Flink 2.x that we are not aware of right now.
> >
> > Best,
> >
> > Konstantin
> >
> > Am Di., 25. Juli 2023 um 12:48 Uhr schrieb Jing Ge
> > :
> >
> > > Hi all,
> > >
> > > Overall, it is a good idea to provide the LTS release, but I'd like to
> > > reference a concrete case as an example to understand what restrictions
> > the
> > > LTS should have.
> > >
> > > Hypothetically, Source-/Sink- Function have been deprecated in 1.x LTS
> > and
> > > removed in 2.0 and the issues[1] are not solved in 2.0. This is a
> typical
> > > scenario that the old APIs are widely used in 1.x LTS and the new APIs
> in
> > > 2.0 are not ready yet to take over all users. We will have the
> following
> > > questions:
> > >
> > > 1. Is this scenario allowed at all? Do we all agree that there could be
> > > some features/functionalities that only work in 1.x LTS after 2.0 has
> > been
> > > released?
> > > 2. How long are we going to support 1.x LTS? 1 year? 2 years? As long
> as
> > > the issues that block users from migrating to 2.0 are not solved, we
> > can't
> > > stop the LTS support, even if the predefined support time expires.
> > > 3. What is the intention to release a new version with (or without)
> LTS?
> > Do
> > > we still want to engage users to migrate to the new release asap? If
> the
> > > old APIs 1.x LTS offer more than the new APIs in 2.0 or it is almost
> > > impossible to migrate, double effort will be required to maintain those
> > > major releases for a very long time. We will be facing many cohorts.
> > >
> > > IMHO, we should be clear with those questions before we start talking
> > about
> > > LTS. WDYT?
> >

Re: [VOTE] Release 2.0 must-have work items

[Konstantin Knauf]

Hi everyone,

I'd just like to add that we also said, that we would continue the
discussion to come up and agree on a list of concrete blockers for the
removal of SourceFunction, so that don't need to have the same discussion
again in half a year. And while we are add it, we should do the same thing
for SinkFunction.

Best,

Konstantin

Am Mi., 26. Juli 2023 um 03:35 Uhr schrieb Xintong Song <
tonysong...@gmail.com>:

> Thanks Leonard for driving this, and thanks everyone for the discussion.
> The back-and-force reflects the importance and complexity around this
> topic. Glad to see we finally reached consensus.
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jul 26, 2023 at 12:42 AM Jing Ge  wrote:
>
> > Thanks Leonard for driving it. We are now on the same page.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Jul 25, 2023 at 9:19 PM Leonard Xu  wrote:
> >
> >> We’ve detailed offline discussions with @Alexander and @Jingsong, about
> >> “Remove SourceFunction” item, we’ve reached a consensus as following:
> >>
> >> 1. Deprecate SourceFunction in 1.18 and implement following improvement
> >> subtasks of FLINK-28045[1] later is reasonable for all of us.
> >>
> >> 2. Deleting SourceFunction API depends on future’s work progress, thus
> >> “Remove SourceFunction APIs” should be a nice to have item. Alexander
> has
> >> volunteered to take these subtasks and would try to finish them next,
> >> thanks again.
> >>
> >> 3. As a nice to have item, and its READY status depends on  future’s
> work
> >> progress,  this won't block release 2.0 must-have item vote.
> >>
> >> Thanks again @Alexander, @Jingsong  and @Xintong for driving these
> things
> >> forward.
> >>
> >> Also CC RMs for 1.18 @QingSheng @Jing @Martijn @Konstantin, I’ve
> >> communicated with Alexander and would like to help review the
> deprecation
> >> PR again.
> >>
> >> Best,
> >> Leonard
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-28045
> >>
> >>
> >> On Jul 21, 2023, at 6:09 PM, Chesnay Schepler 
> wrote:
> >>
> >> On 21/07/2023 11:45, Leonard Xu wrote:
> >>
> >> In this way, the user will see the deprecated API firstly but they can
> >> not find a candidate if we can not finish all tasks in one minor
> version .
> >>
> >>
> >> i'm not convinced that this matters. There will be a whole bunch of APIs
> >> deprecated in 1.18 (that will remain in 1.x!) without a replacement so
> we
> >> can remove them in 2.0.
> >> We already accepted this scenario.
> >>
> >>
> >>
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE] Release 2.0 must-have work items - Round 2

[Konstantin Knauf]

Hi Xingtong,

yes, I am fine with the conclusion for SourceFunction. I chatted with
Leonard a bit last night. Let's continue this vote.

Thanks for the clarification,

Konstantin



Am Mi., 26. Juli 2023 um 04:03 Uhr schrieb Xintong Song <
tonysong...@gmail.com>:

> Hi Konstantin,
>
> It seems the offline discussion has already taken place [1], and part of
> the outcome is that removal of SourceFunction would be a *nice-to-have*
> item for release 2.0 which may not block this *must-have* vote. Do you have
> different opinions about the conclusions in [1]?
>
> If there are still concerns, and the discussion around this topic needs to
> be continued, then I'd suggest (as I mentioned in [2]) not to further block
> this vote (i.e. the decision on other must-have items). Release 2.0 still
> has a long way to go, and it is likely we need to review and update the
> list every once in a while. We can update the list with another vote if
> later we decide to add the removal of SourceFunction to the must-have list.
>
> WDYT?
>
> Best,
>
> Xintong
>
>
> [1] https://lists.apache.org/thread/yyw52k45x2sp1jszldtdx7hc98n72w7k
> [2] https://lists.apache.org/thread/j5d5022ky8k5t088ffm03727o5g9x9jr
>
> On Tue, Jul 25, 2023 at 8:49 PM Konstantin Knauf 
> wrote:
>
> > I assume this vote includes a decision to not removing
> > SourceFunction/SinkFunction in Flink 2.0 (as it has been removed from the
> > table). If this is the case, I don't think, this discussion has
> concluded.
> > There are multiple contributors like myself, Martijn, Alex Fedulov and
> > Maximilian Michels, who have indicated they would be in favor of
> > deprecating/dropping them. This Source/Sink Function discussion seems to
> go
> > in circles in general. I am wondering if it makes sense to have a call
> > about this instead of repeating mailing list discussions.
> >
> > Am Di., 25. Juli 2023 um 13:38 Uhr schrieb Yu Li :
> >
> > > +1 (binding)
> > >
> > > Thanks for driving this, Xintong!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Sun, 23 Jul 2023 at 18:28, Yuan Mei  wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Thanks for driving the discussion through and for all the efforts in
> > > > resolving the complexities :-)
> > > >
> > > > Best
> > > > Yuan
> > > >
> > > > On Thu, Jul 20, 2023 at 5:23 PM Xintong Song 
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start another round of VOTE for the must-have work
> items
> > > for
> > > > > release 2.0 [1]. The corresponding discussion thread is [2], and
> the
> > > > > previous voting thread is [3]. All comments from the previous
> voting
> > > > thread
> > > > > have been addressed.
> > > > >
> > > > > Please note that once the vote is approved, any changes to the
> > > must-have
> > > > > items (adding / removing must-have items, changing the priority)
> > > requires
> > > > > another vote. Assigning contributors / reviewers, updating
> > > descriptions /
> > > > > progress, changes to nice-to-have items do not require another
> vote.
> > > > >
> > > > > The vote will be open until at least July 25, following the
> consensus
> > > > > voting process. Votes of PMC members are binding.
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> > > > >
> > > > > [2]
> https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4
> > > > >
> > > > > [3]
> https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m
> > > > >
> > > >
> > >
> >
> >
> > --
> > https://twitter.com/snntrable
> > https://github.com/knaufk
> >
>


-- 
*Konstantin Knauf*
knauf.konstan...@gmail.com

Re: [VOTE] Release 2.0 must-have work items - Round 2

[Konstantin Knauf]

I assume this vote includes a decision to not removing
SourceFunction/SinkFunction in Flink 2.0 (as it has been removed from the
table). If this is the case, I don't think, this discussion has concluded.
There are multiple contributors like myself, Martijn, Alex Fedulov and
Maximilian Michels, who have indicated they would be in favor of
deprecating/dropping them. This Source/Sink Function discussion seems to go
in circles in general. I am wondering if it makes sense to have a call
about this instead of repeating mailing list discussions.

Am Di., 25. Juli 2023 um 13:38 Uhr schrieb Yu Li :

> +1 (binding)
>
> Thanks for driving this, Xintong!
>
> Best Regards,
> Yu
>
>
> On Sun, 23 Jul 2023 at 18:28, Yuan Mei  wrote:
>
> > +1 (binding)
> >
> > Thanks for driving the discussion through and for all the efforts in
> > resolving the complexities :-)
> >
> > Best
> > Yuan
> >
> > On Thu, Jul 20, 2023 at 5:23 PM Xintong Song 
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start another round of VOTE for the must-have work items
> for
> > > release 2.0 [1]. The corresponding discussion thread is [2], and the
> > > previous voting thread is [3]. All comments from the previous voting
> > thread
> > > have been addressed.
> > >
> > > Please note that once the vote is approved, any changes to the
> must-have
> > > items (adding / removing must-have items, changing the priority)
> requires
> > > another vote. Assigning contributors / reviewers, updating
> descriptions /
> > > progress, changes to nice-to-have items do not require another vote.
> > >
> > > The vote will be open until at least July 25, following the consensus
> > > voting process. Votes of PMC members are binding.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> > >
> > > [2] https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4
> > >
> > > [3] https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m
> > >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

[jira] [Created] (FLINK-32671) Document Externalized Declarative Resource Management

[Konstantin Knauf (Jira)]

Konstantin Knauf created FLINK-32671:


 Summary: Document Externalized Declarative Resource Management
 Key: FLINK-32671
 URL: https://issues.apache.org/jira/browse/FLINK-32671
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-348: Support System Columns in SQL and Table API

[Konstantin Knauf]

Hi Timo,

this makes sense to me. Option 3 seems reasonable, too.

Cheers,

Konstantin

Am Di., 25. Juli 2023 um 12:53 Uhr schrieb Timo Walther :

> Hi everyone,
>
> I would like to start a discussion about introducing the concept of
> "System Columns" in SQL and Table API.
>
> The subject sounds bigger than it actually is. Luckily, Flink SQL
> already exposes the concept of metadata columns. And this proposal is
> just a slight adjustment for how metadata columns can be used as system
> columns.
>
> The biggest problem of metadata columns currently is that a catalog
> implementation can't provide them by default because they would affect
> `SELECT *` when adding another one.
>
> Looking forward to your feedback on FLIP-348:
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-348%3A+Support+System+Columns+in+SQL+and+Table+API
>
> Thanks,
> Timo
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

[Konstantin Knauf]

Hi Jing,

let's not overindex on the Source-/SinkFunction discussion in this thread.

We will generally drop/break a lot of APIs in Flink 2.0. So, naturally
users will need to make more changes to their code in order to migrate from
1.x to Flink 2.0. In order to give them more time to do this, we support
the last Flink 1.x release for a longer time with bug fix releases.

Of course, we still encourage users to migrate to Flink 2.0, because at
some point, we will stop support Flink 1.x. For example, if we followed
Marton's proposal we would support Flink 1.x LTS for about 2 years (roughly
4 minor release cycles) instead of about 1 year (2 minor release cycles)
for regular minor releases. This seems like a reasonable timeframe to me.
It also gives us more time to discover and address blockers in migrating to
Flink 2.x that we are not aware of right now.

Best,

Konstantin

Am Di., 25. Juli 2023 um 12:48 Uhr schrieb Jing Ge
:

> Hi all,
>
> Overall, it is a good idea to provide the LTS release, but I'd like to
> reference a concrete case as an example to understand what restrictions the
> LTS should have.
>
> Hypothetically, Source-/Sink- Function have been deprecated in 1.x LTS and
> removed in 2.0 and the issues[1] are not solved in 2.0. This is a typical
> scenario that the old APIs are widely used in 1.x LTS and the new APIs in
> 2.0 are not ready yet to take over all users. We will have the following
> questions:
>
> 1. Is this scenario allowed at all? Do we all agree that there could be
> some features/functionalities that only work in 1.x LTS after 2.0 has been
> released?
> 2. How long are we going to support 1.x LTS? 1 year? 2 years? As long as
> the issues that block users from migrating to 2.0 are not solved, we can't
> stop the LTS support, even if the predefined support time expires.
> 3. What is the intention to release a new version with (or without) LTS? Do
> we still want to engage users to migrate to the new release asap? If  the
> old APIs 1.x LTS offer more than the new APIs in 2.0 or it is almost
> impossible to migrate, double effort will be required to maintain those
> major releases for a very long time. We will be facing many cohorts.
>
> IMHO, we should be clear with those questions before we start talking about
> LTS. WDYT?
>
> Best regards,
> Jing
>
>
> [1] https://lists.apache.org/thread/734zhkvs59w2o4d1rsnozr1bfqlr6rgm
>
> On Tue, Jul 25, 2023 at 6:08 PM Márton Balassi 
> wrote:
>
> > Hi team,
> >
> > +1 for supporting the last 1.x for a longer than usual period of time and
> > limiting it to bugfixes. I would suggest supporting it for double the
> usual
> > amount of time (4 minor releases).
> >
> > On Tue, Jul 25, 2023 at 9:25 AM Konstantin Knauf 
> > wrote:
> >
> > > Hi Alex,
> > >
> > > yes, I think, it makes sense to support the last 1.x release longer
> than
> > > usual. This should be limited to bugfixes in my opinion.
> > >
> > > Best,
> > >
> > > Konstantin
> > >
> > > Am Di., 25. Juli 2023 um 07:07 Uhr schrieb Xintong Song <
> > > tonysong...@gmail.com>:
> > >
> > > > Hi Alex,
> > > >
> > > > Providing a longer supporting period for the last 1.x minor release
> > makes
> > > > sense to me.
> > > >
> > > > I think we need to be more specific about what LTS means here.
> > > >
> > > >- IIUC, that means for the last 1.x minor release, we will keep
> > > >providing 1.x.y / 1.x.z bugfix release. This is a stronger support
> > > > compared
> > > >to regular minor releases which by default are only supported for
> 2
> > > > minor
> > > >release cycles.
> > > >- Do we only provide bug fixes for the LTS release, or do we also
> > > allow
> > > >backporting features to that release?
> > > >- How long exactly shall we support the LTS release?
> > > >
> > > > And maybe we can make this a general convention for last minor
> releases
> > > for
> > > > all major releases, rather than only discuss it for the 2.0 version
> > bump.
> > > >
> > > > @Leonard,
> > > >
> > > > I'd like to clarify that there are no community decisions yet on
> > release
> > > > 2.0 after 1.19. It is possible to have 1.20 before 2.0.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Tue, Jul 25, 2023 at 11:54 AM Leonard Xu 
> wrote:
> 

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

[Konstantin Knauf]

Hi Alex,

yes, I think, it makes sense to support the last 1.x release longer than
usual. This should be limited to bugfixes in my opinion.

Best,

Konstantin

Am Di., 25. Juli 2023 um 07:07 Uhr schrieb Xintong Song <
tonysong...@gmail.com>:

> Hi Alex,
>
> Providing a longer supporting period for the last 1.x minor release makes
> sense to me.
>
> I think we need to be more specific about what LTS means here.
>
>- IIUC, that means for the last 1.x minor release, we will keep
>providing 1.x.y / 1.x.z bugfix release. This is a stronger support
> compared
>to regular minor releases which by default are only supported for 2
> minor
>release cycles.
>- Do we only provide bug fixes for the LTS release, or do we also allow
>backporting features to that release?
>- How long exactly shall we support the LTS release?
>
> And maybe we can make this a general convention for last minor releases for
> all major releases, rather than only discuss it for the 2.0 version bump.
>
> @Leonard,
>
> I'd like to clarify that there are no community decisions yet on release
> 2.0 after 1.19. It is possible to have 1.20 before 2.0.
>
> Best,
>
> Xintong
>
>
>
> On Tue, Jul 25, 2023 at 11:54 AM Leonard Xu  wrote:
>
> > +1, it’s pretty necessary especially we deprecated so many APIs in 1.18
> > and plan to remove in 2.0.
> >
> > The 1.19 should be a proper version for LTS Release.
> >
> > Best,
> > Leonard
> >
> >
> > > On Jul 25, 2023, at 3:30 AM, Alexander Fedulov <
> > alexander.fedu...@gmail.com> wrote:
> > >
> > > Hello everyone,
> > >
> > > Recently, there were a lot of discussions about the deprecation of
> > various
> > > APIs for the upcoming 2.0 release. It appears there are two main
> > motivations
> > > with opposing directions, causing these discussions to remain
> unsettled.
> > On
> > > one hand, there's a desire to finally trim a wide range of legacy APIs,
> > some
> > > lingering around since the beginning of the 1.x release line (as far
> > back as
> > > 2016). On the other hand, there is a commitment to uphold our
> guarantees
> > to
> > > the users, ensuring a smooth transition.
> > >
> > > I believe we could reconcile these two motivations. My proposition is
> to
> > > designate the final release of the 1.x timeline as a Long-Term Support
> > (LTS)
> > > release. By doing so, we would:
> > >
> > > 1. Enable more efficient cleanup and be liberated to introduce more
> > breaking
> > >   changes, paving the way for greater innovation in the 2.0 release.
> > > 2. Sustain a positive user experience by granting enough time for the
> > > changes
> > >   introduced in 2.0 to stabilize, allowing users to confidently
> > transition
> > >   their production code to the new release.
> > >
> > > I look forward to hearing your thoughts on this proposal.
> > >
> > > Best Regards,
> > > Alex
> >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE][2.0] FLIP-340: Remove rescale REST endpoint

[Konstantin Knauf]

+1 (binding)

Am Mo., 24. Juli 2023 um 14:15 Uhr schrieb Martijn Visser <
martijnvis...@apache.org>:

> +1 (binding)
>
> On Mon, Jul 24, 2023 at 1:10 PM Chesnay Schepler 
> wrote:
>
> > Hello,
> >
> > I'd like to start a vote on FLIP-340.
> >
> > Discussion thread:
> > https://lists.apache.org/thread/zkslk0qzttwgs8j3s951rht3v1tsyqqk
> > FLIP:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-340%3A+Remove+rescale+REST+endpoint
> >
> > Regards,
> > Chesnay
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE][2.0] FLIP-336: Remove "now" timestamp field from REST responses

[Konstantin Knauf]

+1 (binding)

Am Mo., 24. Juli 2023 um 14:15 Uhr schrieb Martijn Visser <
martijnvis...@apache.org>:

> +1 (binding)
>
> On Mon, Jul 24, 2023 at 1:08 PM Chesnay Schepler 
> wrote:
>
> > Hello,
> >
> > I'd like to start a vote on FLIP-336.
> >
> > Discussion thread:
> > https://lists.apache.org/thread/ms3sk0p21n7q2oq0fjtq43koqj2pmwv4
> > FLIP:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424789
> >
> > Regards,
> > Chesnay
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

[jira] [Created] (FLINK-32468) Replace Akka by Pekko

[Konstantin Knauf (Jira)]

Konstantin Knauf created FLINK-32468:


 Summary: Replace Akka by Pekko
 Key: FLINK-32468
 URL: https://issues.apache.org/jira/browse/FLINK-32468
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Konstantin Knauf
 Fix For: 1.18.0


Akka 2.6.x will not receive security fixes from September 2023 onwards (see 
https://discuss.lightbend.com/t/2-6-x-maintenance-proposal/9949). 

A mid-term plan to replace Akka is described in FLINK-29281. In the meantime, 
we suggest to replace Akka by Apache Pekko (incubating), which is a fork of 
Akka 2.6.x under the Apache 2.0 license. This way - if needed - we at least 
have the ability to release security fixes ourselves in collaboration with the 
Pekko community. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-kafka, release candidate #1

[Konstantin Knauf]

+1. Thanks, Gordon!

Am Mo., 3. Apr. 2023 um 06:37 Uhr schrieb Tzu-Li (Gordon) Tai <
tzuli...@apache.org>:

> Hi Martijn,
>
> Since this RC vote was opened, we had three critical bug fixes that was
> merged for the Kafka connector:
>
>- https://issues.apache.org/jira/browse/FLINK-31363
>- https://issues.apache.org/jira/browse/FLINK-31305
>- https://issues.apache.org/jira/browse/FLINK-31620
>
> Given the severity of these issues (all of them are violations of
> exactly-once semantics), and the fact that they are currently not included
> yet in any released version, do you think it makes sense to cancel this RC
> in favor of a new one that includes these?
> Since this RC vote has been stale for quite some time already, it doesn't
> seem like we're throwing away too much effort that has already been done if
> we start a new RC with these critical fixes included.
>
> What do you think?
>
> Thanks,
> Gordon
>
> On Thu, Feb 9, 2023 at 3:26 PM Tzu-Li (Gordon) Tai 
> wrote:
>
> > +1 (binding)
> >
> > - Verified legals (license headers and root LICENSE / NOTICE file).
> AFAICT
> > no dependencies require explicit acknowledgement in the NOTICE files.
> > - No binaries in staging area
> > - Built source with tests
> > - Verified signatures and hashes
> > - Web PR changes LGTM
> >
> > Thanks Martijn!
> >
> > Cheers,
> > Gordon
> >
> > On Mon, Feb 6, 2023 at 6:12 PM Mason Chen 
> wrote:
> >
> >> That makes sense, thanks for the clarification!
> >>
> >> Best,
> >> Mason
> >>
> >> On Wed, Feb 1, 2023 at 7:16 AM Martijn Visser  >
> >> wrote:
> >>
> >> > Hi Mason,
> >> >
> >> > Thanks, [4] is indeed a copy-paste error and you've made the right
> >> > assumption that
> >> >
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapacheflink-1582/org/apache/flink/
> >> > is the correct maven central link.
> >> >
> >> > I think we should use FLINK-30052 to move the Kafka connector code
> from
> >> the
> >> > 1.17 release also over the Kafka connector repo (especially since
> >> there's
> >> > now a v3.0 branch for the Kafka connector, so it can be merged in
> main).
> >> > When those commits have been merged, we can make a next Kafka
> connector
> >> > release (which is equivalent to the 1.17 release, which can only be
> done
> >> > when 1.17 is done because of the split level watermark alignment) and
> >> then
> >> > FLINK-30859 can be finished.
> >> >
> >> > Best regards,
> >> >
> >> > Martijn
> >> >
> >> > Op wo 1 feb. 2023 om 09:16 schreef Mason Chen  >:
> >> >
> >> > > +1 (non-binding)
> >> > >
> >> > > * Verified hashes and signatures
> >> > > * Verified no binaries
> >> > > * Verified LICENSE and NOTICE files
> >> > > * Verified poms point to 3.0.0-1.16
> >> > > * Reviewed web PR
> >> > > * Built from source
> >> > > * Verified git tag
> >> > >
> >> > > I think [4] your is a copy-paste error and I did all the
> verification
> >> > > assuming that
> >> > >
> >> > >
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapacheflink-1582/org/apache/flink/
> >> > > is the correct maven central link.
> >> > >
> >> > > Regarding the release notes, should we close
> >> > > https://issues.apache.org/jira/browse/FLINK-30052 and link it
> there?
> >> > I've
> >> > > created https://issues.apache.org/jira/browse/FLINK-30859 to remove
> >> the
> >> > > existing code from the master branch.
> >> > >
> >> > > Best,
> >> > > Mason
> >> > >
> >> > > On Tue, Jan 31, 2023 at 6:23 AM Martijn Visser <
> >> martijnvis...@apache.org
> >> > >
> >> > > wrote:
> >> > >
> >> > > > Hi everyone,
> >> > > > Please review and vote on the release candidate #1 for
> >> > > > flink-connector-kafka version 3.0.0, as follows:
> >> > > > [ ] +1, Approve the release
> >> > > > [ ] -1, Do not approve the release (please provide specific
> >> comments)
> >> > > >
> >> > > > Note: this is the same code as the Kafka connector for the Flink
> >> 1.16
> >> > > > release.
> >> > > >
> >> > > > The complete staging area is available for your review, which
> >> includes:
> >> > > > * JIRA release notes [1],
> >> > > > * the official Apache source release to be deployed to
> >> dist.apache.org
> >> > > > [2],
> >> > > > which are signed with the key with fingerprint
> >> > > > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
> >> > > > * all artifacts to be deployed to the Maven Central Repository
> [4],
> >> > > > * source code tag v3.0.0-rc1 [5],
> >> > > > * website pull request listing the new release [6].
> >> > > >
> >> > > > The vote will be open for at least 72 hours. It is adopted by
> >> majority
> >> > > > approval, with at least 3 PMC affirmative votes.
> >> > > >
> >> > > > Thanks,
> >> > > > Release Manager
> >> > > >
> >> > > > [1]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352577
> >> > > > [2]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-kafka-3.0.0-rc1
> >> > > > [3] 

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

[Konstantin Knauf]

Congrats!

Am Mo., 27. März 2023 um 14:32 Uhr schrieb Leonard Xu :

> Congratulations!
>
>
> Best,
> Leonard
>
> On Mar 27, 2023, at 5:23 PM, Yu Li  wrote:
>
> Dear Flinkers,
>
>
>
> As you may have noticed, we are pleased to announce that Flink Table Store 
> has joined the Apache Incubator as a separate project called Apache 
> Paimon(incubating) [1] [2] [3]. The new project still aims at building a 
> streaming data lake platform for high-speed data ingestion, change data 
> tracking and efficient real-time analytics, with the vision of supporting a 
> larger ecosystem and establishing a vibrant and neutral open source community.
>
>
>
> We would like to thank everyone for their great support and efforts for the 
> Flink Table Store project, and warmly welcome everyone to join the 
> development and activities of the new project. Apache Flink will continue to 
> be one of the first-class citizens supported by Paimon, and we believe that 
> the Flink and Paimon communities will maintain close cooperation.
>
>
> 亲爱的Flinkers，
>
>
> 正如您可能已经注意到的，我们很高兴地宣布，Flink Table Store 已经正式加入 Apache
> 孵化器独立孵化 [1] [2] [3]。新项目的名字是
> Apache 
> Paimon(incubating)，仍致力于打造一个支持高速数据摄入、流式数据订阅和高效实时分析的新一代流式湖仓平台。此外，新项目将支持更加丰富的生态，并建立一个充满活力和中立的开源社区。
>
>
> 在这里我们要感谢大家对 Flink Table Store 项目的大力支持和投入，并热烈欢迎大家加入新项目的开发和社区活动。Apache Flink
> 将继续作为 Paimon 支持的主力计算引擎之一，我们也相信 Flink 和 Paimon 社区将继续保持密切合作。
>
>
> Best Regards,
> Yu (on behalf of the Apache Flink PMC and Apache Paimon PPMC)
>
> 致礼，
> 李钰（谨代表 Apache Flink PMC 和 Apache Paimon PPMC）
>
> [1] https://paimon.apache.org/
> [2] https://github.com/apache/incubator-paimon
> [3] https://cwiki.apache.org/confluence/display/INCUBATOR/PaimonProposal
>
>
>

-- 
https://twitter.com/snntrable
https://github.com/knaufk

[DISCUSS] Planning Flink 1.18

[Konstantin Knauf]

Hi everyone,

"Nach dem Spiel ist vor dem Spiel" [1] aka "There is always a next
release". With the announcement of Flink 1.17, we have entered the next
release cycle and with it comes the usual routine:

- Release Managers

I'd like to volunteer as one of the release managers this time. It has been
good practice to have a team of release managers from different
backgrounds, so please raise you hand if you'd like to volunteer and get
involved.

- Timeline

Flink 1.17 was released yesterday. With a target release cycle of 4 months,
we should aim for a feature freeze around the end of June.

- Collecting Features

While we are assembling the release management team over the next days and
without anticipating the outcome of any process changes the release
management team might want to try out, I think, we can already start
collecting features as usual on
https://cwiki.apache.org/confluence/display/FLINK/1.18+Release.

-  Feedback

As Matthias already mentioned in the release announcement: if there is any
feedback on the release process or wishes for the future, please share it
with any of the Flink 1.17 release managers (Matthias, Martijn, Lenard and
Qingsheng) or simply in this thread here.

Cheers,

Konstantin

-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-299 Pub/Sub Lite Connector

[Konstantin Knauf]

Hi Daniel,

I think, it would be great to have a PubSub Lite Connector in Flink. Before
you put this proposal up for a vote, though, we need feedback from a
Committer who would review and help maintain it going forward. Ideally,
this Committer would guide one or more contributors from Google to
Committership so that Google could step up and maintain Flink's PubSub and
PubSub Lite Connector in the future. For this, it would be good to
understand how you envision the involvement of the PubSub Lite team at
Google.

I am specifically sensitive on this topic, because the PubSub connector has
lacked attention and maintenance for a long time. There was also a very
short-lived interested by Google in the past to contribute a Google PubSub
Connector [1].

Best,

Konstantin

[1] https://issues.apache.org/jira/browse/FLINK-22380

Am Mi., 8. März 2023 um 14:45 Uhr schrieb Etienne Chauchot <
echauc...@apache.org>:

> Hi,
>
> I agree with Ryan, even if clients might be totally different the
> backend technologies are the same so hosting them in the same repo makes
> sense. Similar thinking made us put all the Cassandra related connectors
> in the same cassandra repo.
>
> Etienne
>
> Le 02/03/2023 à 14:43, Daniel Collins a écrit :
> > Hello Ryan,
> >
> > Unfortunately there's not much shared logic between the two- the clients
> > have to look fundamentally different since the Pub/Sub Lite client
> exposes
> > partitions to the split level for repeatable reads.
> >
> > I have no objection to this living in the same repo as the Pub/Sub
> > connector, if this is an easier way forward than setting up a new repo,
> > sounds good to me. The Pub/Sub team is organizationally close to us, and
> is
> > looking into providing more support for the flink connector in the near
> > future.
> >
> > -Daniel
> >
> > On Thu, Mar 2, 2023 at 3:26 AM Ryan Skraba  >
> > wrote:
> >
> >> Hello Daniel!  Quite a while ago, I started porting the Pub/Sub
> connector
> >> (from an existing PR) to the new source API in the new
> >> flink-connector-gcp-pubsub repository [PR2].  As Martijn mentioned,
> there
> >> hasn't been a lot of attention on this connector; any community
> involvement
> >> would be appreciated!
> >>
> >> Instead of considering this a new connector, is there an opportunity
> here
> >> to offer the two variants (Pub/Sub and Pub/Sub Lite) as different
> artifacts
> >> in that same repo?  Is there much common logic that can be shared
> between
> >> the two?  I'm not as familiar as I should be with Lite, but I do recall
> >> that they share many concepts and _some_ dependencies.
> >>
> >> All my best, Ryan
> >>
> >>
> >> On Wed, Mar 1, 2023 at 11:21 PM Daniel Collins
> >> 
> >> wrote:
> >>
> >>> Hello all,
> >>>
> >>> I'd like to start an official discuss thread for adding a Pub/Sub Lite
> >>> Connector to Flink. We've had requests from our users to add flink
> >> support,
> >>> and are willing to maintain and support this connector long term from
> the
> >>> product team.
> >>>
> >>> The proposal is https://cwiki.apache.org/confluence/x/P51bDg, what
> would
> >>> be
> >>> people's thoughts on adding this connector?
> >>>
> >>> -Daniel
> >>>
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE] FLIP-291: Externalized Declarative Resource Management

[Konstantin Knauf]

+1 (binding)

Am Di., 28. Feb. 2023 um 15:02 Uhr schrieb John Roesler :

> Thanks for the FLIP, David!
>
> I’m +1 (non-binding)
>
> -John
>
> On Tue, Feb 28, 2023, at 07:46, David Morávek wrote:
> > Hi Everyone,
> >
> > I want to start the vote on FLIP-291: Externalized Declarative Resource
> > Management [1]. The FLIP was discussed in this thread [2].
> >
> > The goal of the FLIP is to enable external declaration of the resource
> > requirements of a running job.
> >
> > The vote will last for at least 72 hours (Friday, 3rd of March, 15:00
> CET)
> > unless
> > there is an objection or insufficient votes.
> >
> > [1] https://lists.apache.org/thread/b8fnj127jsl5ljg6p4w3c4wvq30cnybh
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-291%3A+Externalized+Declarative+Resource+Management
> >
> > Best,
> > D.
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [ANNOUNCE] Flink project website is now powered by Hugo

[Konstantin Knauf]

Thanks, Martijn. That was a lot of work.

Am Do., 23. Feb. 2023 um 16:33 Uhr schrieb Maximilian Michels <
m...@apache.org>:

> Congrats! Great work. This was a long time in the making!
>
> -Max
>
> On Thu, Feb 23, 2023 at 3:28 PM Martijn Visser 
> wrote:
> >
> > Hi everyone,
> >
> > The project website at https://flink.apache.org is now powered by Hugo
> [1]
> > which is the same system as the documentation.
> >
> > The theme is the same as the documentation website, so there's no
> redesign
> > involved.
> >
> > If you encounter any issues, please create a Jira ticket and feel free to
> > ping me in it.
> >
> > Thanks to all that have been involved with testing and reviewing!
> >
> > Best regards,
> >
> > Martijn
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-22922
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Flink minor version support policy for old releases

[Konstantin Knauf]

Hi Danny,

yes, this makes a lot of sense. We should probably by default just do one
more patch release shortly after the official support window has ended to
flush out all bug fixes.

Cheers,

Konstantin

Am Fr., 17. Feb. 2023 um 17:52 Uhr schrieb Danny Cranmer <
dannycran...@apache.org>:

> Hello all,
>
> As proposed by Matthias in a separate thread [1], I would like to start a
> discussion on changing the policy wording to include the release of bug
> fixes during their support window. Our current policy [2] is to only
> support the latest two minor versions: " If 1.17.x is the current release,
> 1.16.y is the previous minor supported release. Both versions will receive
> bugfixes for critical issues.". However there may be bug fixes that have
> been resolved but not released during their support window. Consider this
> example:
> 1. Current Flink versions are 1.15.3 and 1.16.1
> 2. We fix bugs for 1.15.3
> 3. 1.17.0 is released
> 4. The 1.15 bug fixes will now not be released unless we get an exception
>
> The current process is subject to race conditions between releases. Should
> we upgrade the policy to allow bugfix releases to support issues that were
> resolved during their support window. I propose we update the policy to
> include:
>
> "Upon release of a new Flink minor version, the community will perform one
> final bugfix release for resolved critical/blocker issues in the Flink
> version losing support."
>
> Let's discuss.
>
> Thanks,
> Danny
>
> [1] https://lists.apache.org/thread/019wsqqtjt6h0cb81781ogzldbjq0v48
> [2] https://flink.apache.org/downloads.html#update-policy-for-old-releases
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Release Flink 1.15.4

[Konstantin Knauf]

+1

Am Mi., 15. Feb. 2023 um 15:21 Uhr schrieb Maximilian Michels <
m...@apache.org>:

> +1
>
> On Wed, Feb 15, 2023 at 12:32 PM Danny Cranmer 
> wrote:
> >
> > Hi all,
> >
> > I would like to discuss creating a new 1.15 patch release (1.15.4). The
> > last 1.15 release is over three months old, and since then, 43 tickets
> have
> > been closed [1], of which 13 are blocker/critical [2]. Given that Flink
> > 1.17 is on the horizon, it will be good to release critical bug fixes
> > before 1.15 goes out of support.
> >
> > Additionally we have community demand for a 1.15.4 release [3] to
> address a
> > critical bug FLINK-28695 [4].
> >
> > I volunteer to be the release manager.
> >
> > Thanks,
> > Danny
> >
> > [1]
> >
> https://issues.apache.org/jira/browse/FLINK-29234?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.15.4%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20%20DESC
> > [2]
> >
> https://issues.apache.org/jira/browse/FLINK-30972?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.15.4%20%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC
> > [3] https://lists.apache.org/thread/dlvmwv3b2vgly2y4cqc53npjkdg66tt4
> > [4] https://issues.apache.org/jira/browse/FLINK-28695
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

[Konstantin Knauf]

Hi Steven,

Sink is already deprecated. It was deprecated at the moment where we
introduced SinkV2.

Hi Jark, Hi Dong,

My understanding is the SinkV2 is a workable interface. The most important
connectors have been migrated (Kafka, Filesystem) and more connectors
(OpenSearch, ElasticSearch, Kinesis) use it successfully. To make SinkV2
public, it does not need to have all possible functionality. Public APIs
can be extended. That's what we do all the time. There will also always be
bugs. So, these points can not be categorical blockers to promote the API.

What are the specific issues/tickets that are blocking us? Can we in your
opinion only deprecate it when every single connector in Apache Flink is
migrated already?

In my opinion it is the time to ask users to the migrate their connectors.
More importantly, @Deprecated would signal users not to build new
connectors on SinkFunction. I would arque its also very misleading to users
to not @Deprecated SinkFunction given that is clearly will be deprecated.

Cheers,

Konstantin


Am Mo., 6. Feb. 2023 um 13:26 Uhr schrieb Jark Wu :

> I agree with Dong Lin.
>
> Oracle explains how to use Deprecate API [1]:
>
> You are strongly recommended to use the Javadoc @deprecated tag with
> > appropriate comments explaining how to use the new API. This ensures
> > developers will *have a workable migration path from the old API to the
> > new API*.
>
>
> From a user's perspective, the workable migration path is very important.
> Otherwise, it blurs the semantics of API deprecation. The Flink API's
> compatibility and stability issues in the past left a bad impression on the
> downstream projects. We should be careful when changing and deprecating
> APIs, especially when there are known migration gaps. I think it's a good
> idea to migrate Flink-owned connectors before marking old API deprecated.
> This ensures downstream projects can migrate to new APIs smoothly.
>
> Best,
> Jark
>
> [1]:
>
> https://docs.oracle.com/javase/8/docs/technotes/guides/javadoc/deprecation/deprecation.html
>
> On Mon, 6 Feb 2023 at 10:01, Steven Wu  wrote:
>
> > Regarding the discussion on global committer [1] for sinks with global
> > transactions, there is no consensus on solving that problem in SinkV2.
> Will
> > it require any breaking change in SinkV2?
> >
> > Also will SinkV1 be deprecated too? or it should happen sometime after
> > SinkFunction deprecation?
> >
> > [1] https://lists.apache.org/thread/82bgvlton9olb591bfg2djv0cshj1bxj
> >
> > On Sun, Feb 5, 2023 at 2:14 AM Dong Lin  wrote:
> >
> > > Hi Konstantin,
> > >
> > > Thanks for the comment! Please see my comment inline.
> > >
> > > Cheers,
> > > Dong
> > >
> > > On Sat, Feb 4, 2023 at 2:06 AM Konstantin Knauf 
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > sorry for joining the discussion late.
> > > >
> > > > 1) Is there an option to deprecate SinkFunction in Flink 1.17 while
> > > leaving
> > > > SinkV2 @PublicEvolving in Flink 1.17. We then aim to make SinkV2
> > @Public
> > > in
> > > > and remove SinkFunction in Flink 1.18. @PublicEvolving are intended
> for
> > > > public use. So, I don't see it as a blocker for deprecating
> > SinkFunction
> > > > that we have to make SinkV2 @PublicEvovling. For reference this is
> the
> > > > description of @PublicEvovling:
> > > >
> > > > /**
> > > >  * Annotation to mark classes and methods for public use, but with
> > > > evolving interfaces.
> > > >  *
> > > >  * Classes and methods with this annotation are intended for
> public
> > > > use and have stable behavior.
> > > >  * However, their interfaces and signatures are not considered to be
> > > > stable and might be changed
> > > >  * across versions.
> > > >  *
> > > >  * This annotation also excludes methods and classes with evolving
> > > > interfaces / signatures within
> > > >  * classes annotated with {@link Public}.
> > > >  */
> > > >
> > > >
> > > > Marking SinkFunction @Deprecated would already single everyone to
> move
> > to
> > > > SinkV2, which we as a community, I believe, have a strong interest
> in.
> > > Its
> > > >
> > >
> > > Yes, I also believe we all have this strong interest. I just hope that
> > this
> > > can be done in the best possible way that does not confuse users.
> > >
> > > I probably

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

[Konstantin Knauf]

Hi everyone,

sorry for joining the discussion late.

1) Is there an option to deprecate SinkFunction in Flink 1.17 while leaving
SinkV2 @PublicEvolving in Flink 1.17. We then aim to make SinkV2 @Public in
and remove SinkFunction in Flink 1.18. @PublicEvolving are intended for
public use. So, I don't see it as a blocker for deprecating SinkFunction
that we have to make SinkV2 @PublicEvovling. For reference this is the
description of @PublicEvovling:

/**
 * Annotation to mark classes and methods for public use, but with
evolving interfaces.
 *
 * Classes and methods with this annotation are intended for public
use and have stable behavior.
 * However, their interfaces and signatures are not considered to be
stable and might be changed
 * across versions.
 *
 * This annotation also excludes methods and classes with evolving
interfaces / signatures within
 * classes annotated with {@link Public}.
 */


Marking SinkFunction @Deprecated would already single everyone to move to
SinkV2, which we as a community, I believe, have a strong interest in. Its
almost comical how long the transition from SourceFurnction/SinkFunction to
Source/Sink takes us. At the same time, we leave ourselves the option to to
make small changes to SinkV2 if any problems arise during the migration of
these connector.

I think, we have a bit of a chicken/egg problem here. The pressure for
users and contributors is not high enough to move away from SinkFunction as
long as its not deprecated, but at the same time we need people to migrate
their connectors to see if there are any gaps in SinkV2. I believe, the
combination proposed above could bridge this problem.

2) I don't understand the argument of waiting until some of the
implementations are @Public. How can we make the implementations of the
SinkV2 API @Public without making SinkV2 @Public? All public methods of
SinkV2 are part of every implementation. So to me it actually seems to be
opposite: in order to make any of the implementation @Public we first need
to make the API @Public.

Cheers,

Konstantin

Am Mo., 30. Jan. 2023 um 13:18 Uhr schrieb Dong Lin :

> Hi Martijn,
>
> Thanks for driving this effort to clean-up the Flink codebase!
>
> I like the idea to cleanup Flink codebase to avoid having two Sinks. On the
> other hand, I also thing the concern mentioned by Jing makes sense. In
> addition to thinking in terms of the rule proposed in FLIP-197
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
> >
> (which
> seems to focus mostly on the Flink developers' perspective), it might be
> useful to also think about the story from users' perspective and make sure
> their concerns can be addressed.
>
> Typically, by marking an API as deprecated, we are effectively telling
> users *they should start to migrate to the new API ASAP and we reserve the
> right to remove this API completely in the 1-2 releases*. Then it might be
> reasonable for users to ask questions such as:
>
> 1) Does SinkV2 public API provides all the functionalities needed to
> migrate my existing code from subclassing SinkFunction to subclassing
> SinkV2?
>
> 2) Is the amount of migration work reasonable? If yes, why is a class such
> as HBaseSinkFunction in Flink's own codebase still depending on
> SinkFunction? Maybe Flink developers should eat their own dog food and
> migrate (or deprecate) those classes in the Flink codebase first?
>
> Based on the discussion in this thread, I am not sure we have good answers
> to those questions yet. For the 1st question above, the answer is *no*
> because we already know that the SinkV2 is currently not able to support
> migration for JdbcSink. For the 2nd question above, we know there are many
> non-deprecated class in Flink codebase that are still depending on SinkV2.
> It is probably not nice to users if we tell them to migrate while we know
> there are existing issues that can prevent them from doing so easily.
>
> In order to move forward to deprecate SinkV2, I think it will be super
> useful to first migrate all the connectors managed by Flink community
> (including all externalized connectors) to use SinkV2. This work won't be
> wasted since we need to do this anyway. And it will also give us a chance
> to validate the capabilities of SinkV2 and fix bugs by ourselves as much as
> possible.
>
> What do you think?
>
> Best Regards,
> Dong
>
>
> On Wed, Jan 18, 2023 at 6:52 PM Martijn Visser 
> wrote:
>
> > Hi all,
> >
> > While discussing FLIP-281 [1] the discussion also turned to the
> > SinkFunction and the SinkV2 API. For a broader discussion I'm opening up
> a
> > separate discussion thread.
> >
> > As Yun Tang has mentioned in that discussion thread, it would be a good
> > time to deprecate the SinkFunction to avoid the need to introduce new
> > functions towards (to be) deprecated APIs. Jing rightfully mentioned that
> > it would be confusing to deprecate the SinkFunction if its successor is
> not
> > yet marked as 

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

[Konstantin Knauf]

Hi everyone,

if I am not mistaken of the sinks mentioned by Joao Kafka, Kinesis &
Kinesis already use the Sink2 API. How were those implemented without
exposing the ExecutionConfig?

Best,

Konstantin


Am Mi., 1. Feb. 2023 um 12:28 Uhr schrieb Lijie Wang <
wangdachui9...@gmail.com>:

> +1 for Option 2, if we can abstract an "ReadableExecutionConfig"
> interface(contains all is/get mehtod), and let ExecutionConfig implements
> ReadableExecutionConfig
>
> Best,
> Lijie
>
> João Boto  于2023年1月17日周二 20:39写道：
>
> > Hi all,
> >
> > As establish a read-only contract seems to be consensual approach,
> talking
> > to Lijie we saw two ways for doing this..
> >
> > Option 1: UnmodifiableExecutionConfig that extends ExecutionConfig (just
> > like the UnmodifiableConfiguration)
> > Pros:
> > - we have all the get methods
> > - don't need to change TypeInformation.createSerializer(ExecutionConfig
> > config)
> > Cons:
> > - we have to override 34 methods that modify things..
> > - new methods to ExecutionConfig will need to be override on
> > UnmodifiableExecutionConfig
> >
> >
> > Option 2: UnmodifiableExecutionConfig without extending ExecutionConfig.
> > Pros:
> > - new class so we don't need to override nothing.
> > - modifications to ExecutionConfig don't affect this class
> > Cons:
> > - need to change TypeInformation adding
> > createSerializer(UnmodifiableExecutionConfig config)
> > - need to add all get methods or only what needed (this could be a pros)
> >
> >
> > What option you think is better?
> >
> >
> >
> > On 2023/01/13 14:15:04 Joao Boto wrote:
> > > Hi flink devs,
> > >
> > > I'd like to start a discussion thread for FLIP-287[1].
> > > This comes from an offline discussion with @Lijie Wang, from
> FLIP-239[2]
> > > specially for the sink[3].
> > >
> > > Basically to expose the ExecutionConfig and JobId on
> SinkV2#InitContext.
> > > This  changes are necessary to correct migrate the current sinks to
> > SinkV2
> > > like JdbcSink, KafkaTableSink and so on, that relies on RuntimeContext
> > >
> > > Comments are welcome!
> > > Thanks,
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240880853
> > > [2]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217386271
> > > [3] https://issues.apache.org/jira/browse/FLINK-25421
> > >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [ANNOUNCE] Apache Flink 1.16.1 released

[Konstantin Knauf]

Great. Thanks, Martijn for managing the release.

Am Mi., 1. Feb. 2023 um 20:26 Uhr schrieb Martijn Visser <
martijnvis...@apache.org>:

> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.16.1, which is the first bugfix release for the Apache Flink 1.16
> series.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Please check out the release blog post for an overview of the improvements
> for this bugfix release:
> https://flink.apache.org/news/2023/01/30/release-1.16.1.html
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352344
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Feel free to reach out to the release managers (or respond to this thread)
> with feedback on the release process. Our goal is to constantly improve the
> release process. Feedback on what could be improved or things that didn't
> go so well are appreciated.
>
> Best regards,
>
> Martijn Visser
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: Reworking the Rescale API

[Konstantin Knauf]

Hi Gyula,

if the adaptive scheduler supported active resource managers, would there
be any other blocker to migrate to it? I don't know much about the
implementation-side here, but conceptually once we have session mode
support and each Jobs in a session clusters declaris their desired
parallelism (!=infinity) there shouldn't be a big gap to support active
resource managers. Am I missing something, Chesnay?

Regarding the complexity, I was referring to the procedure that Max
outlines in his ticket around check if slots are available or not and then
triggering scaling operations. The adaptive scheduler already does this and
is more responsive in that regard than an external process would be in my
understanding.

Cheers,

Konstantin



Am Do., 26. Jan. 2023 um 15:05 Uhr schrieb Gyula Fóra :

> Hi Konstantin!
>
> I think the Adaptive Scheduler still will not support Kubernetes Native
> integration and can only be used in standalone mode. This means that the
> operator needs to manage all resources externally, and compute exactly how
> much new slots are needed during rescaling etc.
>
> I think whatever scaling API we build, it should work for both standalone
> and native integration as much as possible. It's not a duplicated effort to
> add it to the standard scheduler as long as the adaptive scheduler does not
> support active resource management.
>
> Also it seems this will not reduce complexity on the operator side, which
> can already do scaling actions by executing an upgrade.
>
> And a side note: the operator supports both native and standalone
> integration (both standard and adaptive scheduler this way) but the bigger
> problem is actually computing the required number of slots and required new
> resources which is much harder than simply using active resource management.
>
> Cheers,
> Gyula
>
> On Thu, Jan 26, 2023 at 2:57 PM Konstantin Knauf 
> wrote:
>
>> Hi Max,
>>
>> it seems to me we are now running in some of the potential duplication of
>> efforts across the standard and adaptive scheduler that Chesnay had
>> mentioned on the original ticket. The issue of having to do a full restart
>> of the Job for rescaling as well as waiting for resources to be available
>> before doing a rescaling operation were some of the main motivations behind
>> introducing the adaptive scheduler. In the adaptive scheduler we can
>> further do things like only to trigger a rescaling operations exactly when
>> a checkpoint was completed to minimize reprocessing. For Jobs with small
>> state size, the downtime during rescaling can already be << 1 second today.
>>
>> Chesnay and David Moravek are currently in the process of drafting two
>> FLIPs that will extend the support of the adaptive scheduler to session
>> mode and will allow clients to change the desired/min/max parallelism of
>> the vertices of a Job during its runtime via the REST API. We currently
>> plan to publish a draft of these FLIPs next week for discussion. Would you
>> consider moving to the adaptive scheduler for the kubernetes operator
>> provided these FLIPs make it? I think, it has the potential to simplify the
>> logic required for rescaling on the operator side quite a bit.
>>
>> Best,
>>
>> Konstantin
>>
>>
>> Am Do., 26. Jan. 2023 um 12:16 Uhr schrieb Maximilian Michels <
>> m...@apache.org>:
>>
>>> Hey ConradJam,
>>>
>>> Thank you for your thoughtful response. It would be great to start
>>> writing
>>> a FLIP for the Rescale API. If you want to take a stab, please go ahead,
>>> I'd be happy to review. I'm sure Gyula or others will also chime in.
>>>
>>> I want to answer your question so we are aligned:
>>>
>>> ● Does scaling work on YARN, or just k8s?
>>> >
>>>
>>> I think it should work for both YARN and K8s. We would have to make
>>> changes
>>> to the drivers (AbstractResourceManagerDriver) which is implemented for
>>> both K8s and YARN. The outlined approach for rescaling does not require
>>> integrating with those systems, just maybe updating how the driver is
>>> used,
>>> so we should be able to make it work across both YARN and K8s.
>>>
>>> ● Rescaling supports Standalone mode?
>>> >
>>>
>>> Yes, I think it should and easily can. We do use a different type of
>>> resource manager (StandaloneResourceManager, not ActiveResourceManager)
>>> but
>>> I think the logic will sit on a higher level where the ResourceManager
>>> implementation is not relevant.
>>>
>>> ● Can we simplify the recovery steps?

Re: [VOTE] Release flink-connector-gcp-pubsub v3.0.0, release candidate #1

[Konstantin Knauf]

+1 (binding)

* checked Maven and source artifact signatures and checksums - OK
* no binaries or packaged dependencies - OK
* checked website changes - Approved.

Am Fr., 20. Jan. 2023 um 15:39 Uhr schrieb Martijn Visser <
martijnvis...@apache.org>:

> Hi everyone,
> Please review and vote on the release candidate #1 for the version 3.0.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2],
> which are signed with the key with fingerprint
> A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag v3.0.0-rc1 [5],
> * website pull request listing the new release [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352589
> [2]
>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-gcp-pubsub-3.0.0-rc1
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1581/
> [5]
>
> https://github.com/apache/flink-connector-gcp-pubsub/releases/tag/v3.0.0-rc1
> [6] https://github.com/apache/flink-web/pull/604
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: Reworking the Rescale API

[Konstantin Knauf]

Hi Max,

it seems to me we are now running in some of the potential duplication of
efforts across the standard and adaptive scheduler that Chesnay had
mentioned on the original ticket. The issue of having to do a full restart
of the Job for rescaling as well as waiting for resources to be available
before doing a rescaling operation were some of the main motivations behind
introducing the adaptive scheduler. In the adaptive scheduler we can
further do things like only to trigger a rescaling operations exactly when
a checkpoint was completed to minimize reprocessing. For Jobs with small
state size, the downtime during rescaling can already be << 1 second today.

Chesnay and David Moravek are currently in the process of drafting two
FLIPs that will extend the support of the adaptive scheduler to session
mode and will allow clients to change the desired/min/max parallelism of
the vertices of a Job during its runtime via the REST API. We currently
plan to publish a draft of these FLIPs next week for discussion. Would you
consider moving to the adaptive scheduler for the kubernetes operator
provided these FLIPs make it? I think, it has the potential to simplify the
logic required for rescaling on the operator side quite a bit.

Best,

Konstantin


Am Do., 26. Jan. 2023 um 12:16 Uhr schrieb Maximilian Michels <
m...@apache.org>:

> Hey ConradJam,
>
> Thank you for your thoughtful response. It would be great to start writing
> a FLIP for the Rescale API. If you want to take a stab, please go ahead,
> I'd be happy to review. I'm sure Gyula or others will also chime in.
>
> I want to answer your question so we are aligned:
>
> ● Does scaling work on YARN, or just k8s?
> >
>
> I think it should work for both YARN and K8s. We would have to make changes
> to the drivers (AbstractResourceManagerDriver) which is implemented for
> both K8s and YARN. The outlined approach for rescaling does not require
> integrating with those systems, just maybe updating how the driver is used,
> so we should be able to make it work across both YARN and K8s.
>
> ● Rescaling supports Standalone mode?
> >
>
> Yes, I think it should and easily can. We do use a different type of
> resource manager (StandaloneResourceManager, not ActiveResourceManager) but
> I think the logic will sit on a higher level where the ResourceManager
> implementation is not relevant.
>
> ● Can we simplify the recovery steps?
> >
>
> For the first version, I would prefer the simple approach of (1) acquiring
> the required slots for rescaling, then (2) trigger a stop with savepoint
> (3) resubmit the job with updated parallelisms. What you have in mind is a
> bit more involved but certainly a great optimization, especially when only
> a fraction of the job state needs to be repartitioned.
>
> Of course, there are many details, such as
> > ● At some point we may not be able to use this kind of hot update, and
> > still need to restart the job, when this happens, we should prevent users
> > from using rescaling requests
> >
>
> I'm curious to learn more about "hot updates". How would we support this in
> Flink? Would we have to support dynamically repartitioning tasks? I don't
> think Flink supports this yet. For now, restarting the job may be the best
> we can do.
>
> ● After rescaling is submitted, when we fail, there should be a rollback
> > mechanism to roll back to the previous degree of parallelism.
> >
>
> This should not be necessary if all the requirements for rescaling, e.g.
> enough task slots, are satisfied by the Rescale API. I'm not even sure
> rolling back is an option because we can't guarantee that a rollback would
> always work.
>
> Thanks,
> Max
>
> On Tue, Jan 24, 2023 at 6:34 AM ConradJam  wrote:
>
> > Hello max
> >
> > Thanks for driving it, I think there is no problem with your previous
> > suggestion of [1] FLINK-30773. Here I just put forward some supplements
> and
> > doubts.I have some suggestions and insights for this
> >
> >  I have experienced the autoscaling of Flink K8S Operator for a part of
> the
> > time. The current method is to stop the job and modify the parallelism,
> > which will interrupt the business for a long time. I think the purpose of
> > modifying Rescaling Api is to better fit cloud native and reduce the
> impact
> > of job scaling downtime.
> >
> > I have tried scaling with less time, and I call this step "hot update
> > parallelism" (if there is an available Slots, there is no need to
> re-deploy
> > the JobManager Or TaskManager on K8S)
> >
> > Around this topic, I raised the *following questions*:
> > ● Does scaling work on YARN, or just k8s?
> >○ I think we can support running on K8S for the first version, and
> Yarn
> > can be considered later
> > ● Rescaling supports Standalone mode?
> >○ I think it can be supported. The essence is just to modify the
> > parallelism of job vertices. As for the tuning strategy, it should be
> > determined by the external system or K8S Operator
> > ● Can we simplify the 

Re: [VOTE] Release 1.16.1, release candidate #1

[Konstantin Knauf]

Thanks, Martijn.

+1 (binding)

- Verified checksums and signatures for binaries & sources - OK
- checked pom changes for newly introduced dependencies - OK
- went over flink-web PR - Looks good except for Matthias' remarks

Am Do., 26. Jan. 2023 um 12:34 Uhr schrieb Sergey Nuyanzin <
snuyan...@gmail.com>:

> Thanks Martijn.
> +1 (non-binding)
>
> - Verified checksums and signatures
> - Built from sources
> - Compared downloaded with mentioned git tag
> - Verified version in poms
> - Ran several queries in batch and streaming (standalone cluster)
> - Verified NOTICE and LICENSE files
>
>
> On Mon, Jan 23, 2023 at 2:06 PM Matthias Pohl
>  wrote:
>
> > Thanks Martijn for pushing this one.
> > +1 (non-binding)
> >
> > * Downloaded artifacts & built Flink from sources
> > * Verified SHA512 checksums & GPG signatures
> > * Compared checkout with provided sources
> > * Verified pom file versions
> > * Went over NOTICE file/pom files changes without finding anything
> > suspicious
> > * Deployed standalone session cluster and ran WordCount example in batch
> > and streaming: Nothing suspicious in log files found
> > * went over flink-web PR
> >
> > On Thu, Jan 19, 2023 at 5:08 PM Martijn Visser  >
> > wrote:
> >
> > > Hi everyone,
> > > Please review and vote on the release candidate #1 for the version
> > 1.16.1,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > >
> > > The complete staging area is available for your review, which includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release and binary convenience releases to
> > be
> > > deployed to dist.apache.org [2], which are signed with the key with
> > > fingerprint A5F3BCE4CBE993573EC5966A65321B8382B219AF [3] (,
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tag "release-1.16.1-rc1" [5],
> > > * website pull request listing the new release and adding announcement
> > blog
> > > post [6].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Release Manager
> > >
> > > NOTE: The maven artifacts have been signed by Chesnay with the key with
> > > fingerprint C2EED7B111D464BA
> > >
> > > [1]
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352344
> > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.16.1-rc1/
> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1580
> > > [5] https://github.com/apache/flink/releases/tag/release-1.16.1-rc1
> > > [6] https://github.com/apache/flink-web/pull/603
> > >
> >
>
>
> --
> Best regards,
> Sergey
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-286: Fix the API stability/scope annotation inconsistency in AbstractStreamOperator

[Konstantin Knauf]

Hi Becket,

> It is a basic rule of public API that anything exposed by a public
interface should also be public.

I agree with this in general. Did you get an overview of where we currently
violate this? Is this something that the Arc42 architecture tests could
test for so that as a first measure we don't introduce more occurrences
(cc @Ingo).

Maybe its justified to make a pass over all of these occurrences and
resolve these occurrences one way or another either making the
members/parameters @PublicEvoling or actually making a class/method
@Internal even if its was @PublicEvoling before. I think, this could be the
better option than having @PublicEvolving classes/methods that really
aren't.

Cheers,

Konstantin

Am Fr., 13. Jan. 2023 um 17:02 Uhr schrieb Becket Qin :

> Hi Dawid,
>
> Thanks for the reply. I am currently looking at the Beam Flink runner, and
> there are quite some hacks the Beam runner has to do in order to deal with
> the backwards incompatible changes in AbstractStreamOperator and some of
> the classes exposed by it. Regardless of what we think, the fact is that
> AbstractStreamOperator is marked as PublicEvolving today, and our users use
> it. It is a basic rule of public API that anything exposed by a public
> interface should also be public. This is the direct motivation of this
> FLIP.
>
> Regarding the StreamTask / StreamConfig exposure, if you look at the
> StreamOperatorFactory which is also a PublicEvolving class, it actually
> exposes the StreamTask, StreamConfig as well as some other classes in the
> StreamOperatorParameters. So these classes are already exposed in multiple
> public APIs.
>
> Keeping our public API stability guarantee is really fundamental and
> critical to the users. With the current status of inconsistent API
> stability annotations, I don't see how can we assure of that. From what I
> can see, accidental backwards incompatible changes is likely going to keep
> happening. So I'd say let's see how to fix forward instead of doing
> nothing.
>
> BTW, Initially I thought this is just an accidental mismatch, but after
> further exam, it looks that it is a bigger issue. I guess one of the
> reasons we end up in this situation is that we haven't really thought it
> through regarding the boundary between framework and user space, i.e. what
> framework primitives we want to expose to the users. So instead we just
> expose a bunch of internal things and hope users only use the "stable" part
> of them.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Fri, Jan 13, 2023 at 7:28 PM Dawid Wysakowicz 
> wrote:
>
> > Hi Becket,
> >
> > May I ask what is the motivation for the change?
> >
> > I'm really skeptical about making any of those classes `Public` or
> > `PublicEvolving`. As far as I am concerned there is no agreement in the
> > community if StreamOperator is part of the `Public(Evolving)` API. At
> > least I think it should not. I understand `AbstractStreamOperator` was
> > marked with `PublicEvolving`, but I am really not convinced it was the
> > right decision.
> >
> > The listed classes are not the only classes exposed to
> > `AbstractStreamOperator` that are `Internal` that break the consistency
> > and I am sure there is no question those should remain `Internal` such
> > as e.g. StreamTask, StreamConfig, StreamingRuntimeContext and many more.
> >
> > As it stands I am strongly against giving any additional guarantees on
> > API stability to the classes there unless there is a good motivation for
> > a given feature and we're sure this is the best way to go forward.
> >
> > Thus I'm inclined to go with -1 on any proposal on changing annotations
> > for the sake of matching the one on `AbstractStreamOperator`. I might be
> > convinced to support requests to give better guarantees for well
> > motivated features that are now internal though.
> >
> > Best,
> >
> > Dawid
> >
> > On 12/01/2023 10:20, Becket Qin wrote:
> > > Hi flink devs,
> > >
> > > I'd like to start a discussion thread for FLIP-286[1].
> > >
> > > As a recap, currently while AbstractStreamOperator is a class marked as
> > > @PublicEvolving, some classes exposed via its methods / fields are
> > > marked as @Internal. This FLIP wants to fix this inconsistency of
> > > stability / scope annotation.
> > >
> > > Comments are welcome!
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240880841
> > >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Release Flink 1.16.1

[Konstantin Knauf]

Hi Martijn,

thanks for bringing it up and +1 to do a release rather soon.

Best,

Konstantin

Am Fr., 16. Dez. 2022 um 06:17 Uhr schrieb ConradJam :

> Hi Martijn,
>
> Thank you for bringing this up. I'm in favor of this proposal, by the way,
> FLINK-30116 has been fixed
>
>
>
> Martijn Visser  于2022年12月16日周五 02:52写道：
>
> > Hi everyone,
> >
> > I would like to open a discussion about releasing Flink 1.16.1. We've
> > released Flink 1.16 at the end of October, but we already have 58 fixes
> > listed for 1.16.1, including a blocker [1] on the environment variables
> and
> > a number of critical issues. Some of the critical issues are related to
> the
> > bugs on the Sink API, on PyFlink and some correctness issues.
> >
> > There are also a number of open issues with a fixVersion set to 1.16.1,
> so
> > it would be good to understand what the community thinks of starting a
> > release or if there are some fixes that should be included with 1.16.1.
> >
> > Best regards,
> >
> > Martijn
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-30116
> >
>
>
> --
> Best
>
> ConradJam
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE] FLIP-277: Glue catalog support in Flink

[Konstantin Knauf]

+1 (binding)

Am Fr., 16. Dez. 2022 um 15:31 Uhr schrieb Dong Lin :

> +1 (binding)
>
> Thanks for driving this!
>
> On Fri, Dec 16, 2022 at 6:17 PM Samrat Deb  wrote:
>
> > Hi All, Thanks for all the feedback so far. Based on the discussion[1] we
> > seem to have a consensus, so I would like to start a vote on FLIP-277[2].
> > The vote will last for at least 72 hours (Dec 21st at 11:00 GMT,
> excluding
> > weekend days) unless there is an objection or insufficient votes.
> >
> > Please  ignore the previous mail.
> >
> > Best, Samrat
> >
> > [1] https://lists.apache.org/thread/d8lhwz1so2l6zsybwntt49xmmkdohdvt
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-277%3A+Native+GlueCatalog+Support+in+Flink
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Externalizing Kafka connector

[Konstantin Knauf]

Hi Mason,

great, thanks a lot for working on this. Will greatly speed up CI of the
core repository once this is finalized.

Cheers

Konstantin


Am Mi., 7. Dez. 2022 um 07:23 Uhr schrieb Mason Chen :

> Hi all,
>
> I've finished the first pass on externalizing the Kafka connector under the
> release-1.16 branch in this
> https://github.com/apache/flink-connector-kafka/pull/1. The docs,
> connectors, and e2e tests have been relocated and CI on my fork has
> been running green. Thanks for everyone else's work that served as a great
> example!
>
> Next, we will work on a 1.16 release and port the release-1.17 changes over
> to the new repo afterwards. Please take a look and let me know if you have
> any questions!
>
> Best,
> Mason
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-277: Native GlueCatalog Support in Flink

[Konstantin Knauf]

Hi Samrat,

+1 to the effort and +1 to adding it to flink-connector-aws.

Can you explain how users are expected to authenticate with AWS Glue? I
don't see any catalog options regardng authx. So I assume the credentials
are taken from the environment?

Best,

Konstantin



Am Fr., 9. Dez. 2022 um 16:08 Uhr schrieb Jark Wu :

> Hi Samrat,
>
> Thanks a lot for driving the new catalog, and sorry for jumping into the
> discussion late.
>
> As Flink SQL is becoming the first-class citizen of the Flink API, we are
> planning to push Catalog
> to become the first-class citizen of the connector instead of Source &
> Sink. For Flink SQL users,
> using Catalog is as natural and user-friendly as working with databases,
> rather than having to define
> DDL and schemas over and over again. This is also how Trino/Presto does.
>
> Regarding the repo for the Glue catalog, I think we can add it to
> flink-connector-aws. We don't need
> separate repos for Catalogs because Catalog is a kind of connector (others
> are sources & sinks).
> For example, MySqlCatalog[1] and PostgresCatalog[2] are in
> flink-connector-jdbc, and HiveCatalog is
> in flink-connector-hive. This can reduce repository maintenance, and I
> think maybe some common
> AWS utils can be shared there.  cc @Danny Cranmer  >
> what do you think about this?
>
> Besides, I have a question about Glue Namespace. Could you share the
> documentation of the Glue
>  Namespaces? (Sorry, I didn't find it.) According to the "Flink Glue
> Metaspace Mapping" section,
> if there is a database "mydb" under namespace "ns1", is that mean the
> database name in Flink is "ns1.mydb"?
>
> Best,
> Jark
>
>
> [1]:
>
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/catalog/MySqlCatalog.java
> [2]:
>
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/catalog/PostgresCatalog.java
>
> On Fri, 9 Dec 2022 at 08:51, Dong Lin  wrote:
>
> > Hi Samrat,
> >
> > Sorry for the late reply. Yeah I am referring to creating a similar
> > external repo such as flink-catalog-glue. flink-connector-aws is already
> > named with `connector` so it seems a bit weird to put a catalog there.
> >
> > Thanks!
> > Dong
> >
> > On Wed, Dec 7, 2022 at 1:04 PM Samrat Deb  wrote:
> >
> > > Hi Dong Lin,
> > >
> > > Since this is the first proposal for adding a vendor-specific catalog
> > > > library in Flink, I think maybe we should also externalize those
> > catalog
> > > > libraries similar to how we are externalizing connector libraries. It
> > is
> > > > likely that we might want to add catalogs for other vectors in the
> > > future.
> > > > Externalizing those catalogs can make Flink development more scalable
> > in
> > > > the long term.
> > >
> > > Initially i mis-interpretted externalising the catalogs, There already
> > > exists an externalised connector for aws [1].
> > > Are you referring to creating a similar external repo for catalogs or
> > will
> > > it be better to add it in flink-connector-aws[1] ?
> > >
> > > [1] https://github.com/apache/flink-connector-aws
> > >
> > > Samrat
> > >
> > > On Tue, Dec 6, 2022 at 6:52 PM Samrat Deb 
> wrote:
> > >
> > > > Hi Dong Lin,
> > > >
> > > > Aws Glue Data catalog is vendor specific and in future we will get
> such
> > > > type of implementation from different providers. We should
> > > > definitely externalize these catalog libraries similar to flink
> > > connectors.
> > > > I am thinking of creating
> > > > flink-catalog similar to flink-connector under the root (flink). glue
> > > > catalog can be one of modules under the flink-catalog . Please
> suggest
> > if
> > > > there is a better structure we can create for catalogs.
> > > >
> > > >
> > > > It is mentioned in the FLIP that there will be two types of
> > SdkHttpClient
> > > >> supported based on the catalog option http-client.type. Is
> > > >> http-client.type
> > > >> a public config for the GlueCatalog? If yes, can we add this config
> to
> > > the
> > > >> "Configurations" section and explain how users should choose the
> > client
> > > >> type?
> > > >
> > > >
> > > > yes http-client.type is public config for the GlueCatalog. By default
> > > > client-type will be `urlconnection` , if user don't specify any
> > > connection
> > > > type.
> > > > I have updated the FLIP-277[1] #configuration section with all the
> > > configs
> > > > . Please review it again .
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-277%3A+Native+GlueCatalog+Support+in+Flink
> > > >
> > > > Samrat
> > > >
> > > > On Tue, Dec 6, 2022 at 5:50 PM Samrat Deb 
> > wrote:
> > > >
> > > >> Hi Yuxia,
> > > >>
> > > >> Thank you for reviewing the flip and putting forward your
> observations
> > > >> and comments.
> > > >>
> > > >> 1: I noticed there's a YAML part in the section of "Using the
> > Catalog",
> > > >>> what do you 

Re: [ANNOUNCE] New Apache Flink PMC Members - Godfrey He, Xingbo Huang

[Konstantin Knauf]

Congrats to both.

Am Mi., 23. Nov. 2022 um 10:45 Uhr schrieb yu zelin :

> Congratulations，Godfrey and Xingbo!
>
> Best,
> Yu Zelin
> > 2022年11月23日 12:18，Dian Fu  写道：
> >
> > Hi everyone,
> >
> > On behalf of the Apache Flink PMC, I'm very happy to announce that
> Godfrey
> > He and Xingbo Huang have joined the Flink PMC!
> >
> > Godfrey He becomes a Flink committer in Sep 2020. His contributions are
> > mainly focused on the Flink table module, covering almost all important
> > parts such as Client(SQL Client, SQL gateway, JDBC driver, etc), API, SQL
> > parser, query optimization, query execution, etc. Especially in the query
> > optimization part, he built the query optimization framework and the cost
> > model, improved the statistics information and made a lot of
> optimizations,
> > e.g. dynamic partition pruning, join hint, multiple input rewrite, etc.
> In
> > addition, he has done a lot of groundwork, such as refactoring the
> > ExecNode(which is the basis for further DAG optimizations) and SQL Plan
> > JSON serialization (which is a big step to support SQL job version
> > upgrade). Besides that, he's also helping the projects in other ways,
> e.g.
> > managing releases, giving talks, etc.
> >
> > Xingbo Huang becomes a Flink committer in Feb 2021. His contributions are
> > mainly focused on the PyFlink module and he's the author of many
> important
> > features in PyFlink, e.g. Cython support, Python thread execution mode,
> > Python UDTF support, Python UDAF support in windowing, etc. He is also
> one
> > of the main contributors in the Flink ML project. Besides that, he's also
> > helping to manage releases, taking care of the build stabilites, etc.
> >
> > Congratulations and welcome them as Apache Flink PMC!
> >
> > Regards,
> > Dian
>
>

-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [ANNOUNCE] New Apache Flink Committer - Matyas Orhidi

[Konstantin Knauf]

Congrats!

Am Di., 22. Nov. 2022 um 11:46 Uhr schrieb Péter Váry <
peter.vary.apa...@gmail.com>:

> Congratulations Mátyás!
>
> On Tue, Nov 22, 2022, 06:40 Matthias Pohl 
> wrote:
>
> > Congratulations, Matyas :)
> >
> > On Tue, Nov 22, 2022 at 11:44 AM Xingbo Huang 
> wrote:
> >
> > > Congrats Matyas!
> > >
> > > Best,
> > > Xingbo
> > >
> > > Yanfei Lei  于2022年11月22日周二 11:18写道：
> > >
> > > > Congrats Matyas! 
> > > >
> > > > Zheng Yu Chen  于2022年11月22日周二 11:15写道：
> > > >
> > > > > Congratulations ~ 
> > > > >
> > > > > Márton Balassi  于2022年11月21日周一 22:18写道：
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > On behalf of the PMC, I'm very happy to announce Matyas Orhidi
> as a
> > > new
> > > > > > Flink
> > > > > > committer.
> > > > > >
> > > > > > Matyas has over a decade of experience of the Big Data ecosystem
> > and
> > > > has
> > > > > > been working with Flink full time for the past 3 years. In the
> open
> > > > > source
> > > > > > community he is one of the key driving members of the Kubernetes
> > > > Operator
> > > > > > subproject. He implemented multiple key features in the operator
> > > > > including
> > > > > > the metrics system and the ability to dynamically configure
> watched
> > > > > > namespaces. He enjoys spreading the word about Flink and
> regularly
> > > does
> > > > > so
> > > > > > via authoring blogposts and giving talks or interviews
> representing
> > > the
> > > > > > community.
> > > > > >
> > > > > > Please join me in congratulating Matyas for becoming a Flink
> > > committer!
> > > > > >
> > > > > > Best,
> > > > > > Marton
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Yanfei
> > > >
> > >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

[jira] [Created] (FLINK-30116) Don't Show Env Vars in Web UI

[Konstantin Knauf (Jira)]

Konstantin Knauf created FLINK-30116:


 Summary: Don't Show Env Vars in Web UI
 Key: FLINK-30116
 URL: https://issues.apache.org/jira/browse/FLINK-30116
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Web Frontend
Affects Versions: 1.16.0
Reporter: Konstantin Knauf
 Fix For: 1.16.1


As discussed and agreed upon in [1], we'd like to revert [2] and not show any 
environment variables in the Web UI for security reasons. 

[1] https://lists.apache.org/thread/rjgk15bqttvblp60zry4n5pw4xjw7q9k 
[2] https://issues.apache.org/jira/browse/FLINK-28311



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Env Vars in the Flink Web UI

[Konstantin Knauf]

Hi everyone,

thanks a lot for your feedback so far. Right now, we have pretty much a
consensus to not show environment variables at all in the Web UI going
forward.

I think, we can address this in 1.16.1, as I consider this a vulnerability
that should be addressed in a patch release rather than waiting for the
next minor release.

Are there any other suggestions on how to proceed?

Cheers,

Konstantin



Am Mi., 16. Nov. 2022 um 09:23 Uhr schrieb Gyula Fóra :

> I am not opposed to removing this completely based on Chesnay's reasoning.
> In general I agree that this feature probably does more harm than good.
>
> Gyula
>
> On Wed, Nov 16, 2022 at 9:13 AM Chesnay Schepler 
> wrote:
>
> > I'm inclined to go with d), removing it entirely.
> >
> > I must admit that I liked the idea behind the change; exposing more
> > information about what might impact Flink's behavior is a good thing,
> > although I'm irked that the statement in the FLIP about env variables
> > already being exposed in the logs just isn't correct.
> >
> > I was quite disappointed when I saw that b) wasn't already implemented.
> > It is concerning that this actually made it's way through the review
> > rounds as-is.
> >
> > That being said I don't think that b) would be sufficient in anyway; the
> > desensitization logic for config options is quite limited (and
> > inherently not perfect), but config options are too important to not log
> > them. This isn't really the case for environment variables that have
> > limited effects on Flink, and it's just too easy to leak secrets.
> > Oh you abbreviated PASSWORD to PW? Well you just leaked it.
> >
> > This brings us to c), where my immediate gut instinct was that no ones
> > gonna bother to do that.
> >
> > as for e*) (opt-in flag that Gyula proposed); I think it's to easy to
> > shoot yourself in the foot somewhere down the line. It may be fine at
> > one point but setups evolve after all, and this seems like something to
> > easily slip through.
> >
> > On 15/11/2022 15:41, Konstantin Knauf wrote:
> > > Hi everyone,
> > >
> > > important correction, this is since 1.16.0, not 1.17+.
> > >
> > > Best,
> > >
> > > Konstantin
> > >
> > > Am Di., 15. Nov. 2022 um 14:25 Uhr schrieb Gyula Fóra <
> > gyula.f...@gmail.com
> > >> :
> > >> Thanks for bringing this important issue to discussion Konstantin!
> > >>
> > >> I am in favor of not showing them by default with an optional
> > configuration
> > >> to enable it.
> > >> Otherwise this poses a big security risk of exposing previously hidden
> > >> information after upgrade.
> > >>
> > >> Gyula
> > >>
> > >> On Tue, Nov 15, 2022 at 2:15 PM Maximilian Michels 
> > wrote:
> > >>
> > >>> Hey Konstantin,
> > >>>
> > >>> I'd be in favor of not printing them at all, i.e. option (d). We have
> > the
> > >>> configuration page which lists the effective config and already
> removes
> > >> any
> > >>> known secrets.
> > >>>
> > >>> -Max
> > >>>
> > >>> On Tue, Nov 15, 2022 at 11:26 AM Konstantin Knauf  >
> > >>> wrote:
> > >>>
> > >>>> Hi all,
> > >>>>
> > >>>> since Flink 1.17 [1] the Flink Web UI prints *all* environment
> > >> variables
> > >>> of
> > >>>> the Taskmanager or Jobmanagers hosts (Jobmanager -> Configuration ->
> > >>>> Environment). Given that environment variables are often used to
> store
> > >>>> sensitive information, I think, it is wrong and dangerous to print
> > >> those
> > >>> in
> > >>>> the Flink Web UI. Specifically, thinking about how Kubernetes
> Secrets
> > >> are
> > >>>> usually injected into Pods.
> > >>>>
> > >>>> One could argue that anyone who can submit a Flink Job to a cluster
> > has
> > >>>> access to these environment variables anyway, but not everyone who
> has
> > >>>> access to the Flink UI can submit a Flink Job.
> > >>>>
> > >>>> I see the the following options:
> > >>>> a) leave as is
> > >>>> b) apply same obfuscation as in flink-conf.yaml based on some
> > heuristic
> > >>> (no
> > >>>> "secret", "password" in env var name)
> > >>>> c) only print allow-listed values
> > >>>> d) don't print any env vars in the web UI (at least by default)
> > >>>>
> > >>>> What do you think?
> > >>>>
> > >>>> Cheers,
> > >>>>
> > >>>> Konstantin
> > >>>>
> > >>>> [1] https://issues.apache.org/jira/browse/FLINK-28311
> > >>>>
> > >>>> --
> > >>>> https://twitter.com/snntrable
> > >>>> https://github.com/knaufk
> > >>>>
> > >
> >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: Env Vars in the Flink Web UI

[Konstantin Knauf]

Hi everyone,

important correction, this is since 1.16.0, not 1.17+.

Best,

Konstantin

Am Di., 15. Nov. 2022 um 14:25 Uhr schrieb Gyula Fóra :

> Thanks for bringing this important issue to discussion Konstantin!
>
> I am in favor of not showing them by default with an optional configuration
> to enable it.
> Otherwise this poses a big security risk of exposing previously hidden
> information after upgrade.
>
> Gyula
>
> On Tue, Nov 15, 2022 at 2:15 PM Maximilian Michels  wrote:
>
> > Hey Konstantin,
> >
> > I'd be in favor of not printing them at all, i.e. option (d). We have the
> > configuration page which lists the effective config and already removes
> any
> > known secrets.
> >
> > -Max
> >
> > On Tue, Nov 15, 2022 at 11:26 AM Konstantin Knauf 
> > wrote:
> >
> > > Hi all,
> > >
> > > since Flink 1.17 [1] the Flink Web UI prints *all* environment
> variables
> > of
> > > the Taskmanager or Jobmanagers hosts (Jobmanager -> Configuration ->
> > > Environment). Given that environment variables are often used to store
> > > sensitive information, I think, it is wrong and dangerous to print
> those
> > in
> > > the Flink Web UI. Specifically, thinking about how Kubernetes Secrets
> are
> > > usually injected into Pods.
> > >
> > > One could argue that anyone who can submit a Flink Job to a cluster has
> > > access to these environment variables anyway, but not everyone who has
> > > access to the Flink UI can submit a Flink Job.
> > >
> > > I see the the following options:
> > > a) leave as is
> > > b) apply same obfuscation as in flink-conf.yaml based on some heuristic
> > (no
> > > "secret", "password" in env var name)
> > > c) only print allow-listed values
> > > d) don't print any env vars in the web UI (at least by default)
> > >
> > > What do you think?
> > >
> > > Cheers,
> > >
> > > Konstantin
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-28311
> > >
> > > --
> > > https://twitter.com/snntrable
> > > https://github.com/knaufk
> > >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Env Vars in the Flink Web UI

[Konstantin Knauf]

Hi all,

since Flink 1.17 [1] the Flink Web UI prints *all* environment variables of
the Taskmanager or Jobmanagers hosts (Jobmanager -> Configuration ->
Environment). Given that environment variables are often used to store
sensitive information, I think, it is wrong and dangerous to print those in
the Flink Web UI. Specifically, thinking about how Kubernetes Secrets are
usually injected into Pods.

One could argue that anyone who can submit a Flink Job to a cluster has
access to these environment variables anyway, but not everyone who has
access to the Flink UI can submit a Flink Job.

I see the the following options:
a) leave as is
b) apply same obfuscation as in flink-conf.yaml based on some heuristic (no
"secret", "password" in env var name)
c) only print allow-listed values
d) don't print any env vars in the web UI (at least by default)

What do you think?

Cheers,

Konstantin

[1] https://issues.apache.org/jira/browse/FLINK-28311

-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: ASF Slack

[Konstantin Knauf]

Hi everyone,

I agree with Xintong in the sense that I don't see what has changed since
the original decision on this topic. In my opinion, there is a high cost in
moving to ASF now, namely I fear we will loose many of the >1200 members
and the momentum that I see in the workspace. To me there would need to be
strong reason for reverting this decision now.

Cheers,

Kosntantin

Am Di., 8. Nov. 2022 um 10:35 Uhr schrieb Xintong Song <
tonysong...@gmail.com>:

> Hi Max,
>
> Thanks for bringing this up. I'm open to a re-evaluation of the Slack
> usages.
>
> In the initial discussion for creating the Slack workspace [1], leveraging
> the ASF Slack was indeed brought up as an alternative by many folks. I
> think we have chosen a dedicated Flink Slack over the ASF Slack mainly for
> two reasons.
> - ASF Slack is limited to people with an @apache.org email address
> - With a dedicated Flink Slack, we have the full authority to manage and
> customize it. E.g., archiving / removing improper channels, reporting the
> build and benchmark reports to channels, subscribing and re-post Flink blog
> posts.
> As far as I can see, these concerns for the ASF slack have not changed
> since the previous decision.
>
> On the other hand, could you explain a bit more about what are the problems
> / drawbacks that you see in the current Flink Slack?
> - I assume having to join too many workspaces counts one
>
> Best,
>
> Xintong
>
>
> [1] https://lists.apache.org/thread/n43r4qmwprhdmzrj494dbbwr9w7bbdcv
>
> On Tue, Nov 8, 2022 at 4:51 PM Martijn Visser 
> wrote:
>
> > If you click on the link from Beam via an incognito window/logged out of
> > Slack, you will be prompted to provide the workspace URL of the ASF. If
> you
> > do that, you're prompted for a login screen or you can create an account.
> > Creating an account prompts you to have an @apache.org email address.
> See
> > https://imgur.com/a/jXvr5Ai
> >
> > So for me that's a -1 for switching to the ASF workspace.
> >
> > On Mon, Nov 7, 2022 at 10:52 PM Austin Bennett 
> wrote:
> >
> > > +1 to leveraging the larger ASF Community/Resources Slack Channel
> rather
> > > than an independant one ... ASSUMING ANYONE CAN JOIN [ so that needs to
> > be
> > > verified ].
> > >
> > > On Mon, Nov 7, 2022 at 9:05 AM Maximilian Michels 
> > wrote:
> > >
> > >> There is a way to work around the invite issue. For example, the Beam
> > >> project has a direct invite link which sends you to the #beam channel:
> > >> https://app.slack.com/client/T4S1WH2J3/C9H0YNP3P I'm not 100% sure
> > >> whether
> > >> this link actually works. I've take it from:
> > >> https://beam.apache.org/community/join-beam/
> > >>
> > >> -Max
> > >>
> > >> On Fri, Nov 4, 2022 at 1:48 PM Martijn Visser <
> martijnvis...@apache.org
> > >
> > >> wrote:
> > >>
> > >> > Hi Max,
> > >> >
> > >> > > I wonder how that has played out since the creation of the Slack
> > >> > workspace? I haven't been following the Slack communication.
> > >> >
> > >> > I think it has primarily played a role next to the existing User
> > mailing
> > >> > list: lots of user questions. There were maybe a handful of
> > >> conversations
> > >> > where the result of the conversation was a request to open a Jira,
> > >> create a
> > >> > FLIP or open up a discussion on the Dev list.
> > >> >
> > >> > > There is an invite link that they can use. Maybe not the most
> > >> > straight-forward process but I think it doesn't stop users from
> > joining.
> > >> >
> > >> > That's only possible to use for those with an @apache.org e-mail
> > >> address,
> > >> > see https://infra.apache.org/slack.html. Anyone else would need to
> be
> > >> > invited by a committer, but I wouldn't be in favour of needing to
> > spend
> > >> > committers time in adding/inviting members on an ASF Slack instance.
> > >> >
> > >> > Best regards,
> > >> >
> > >> > Martijn
> > >> >
> > >> > On Fri, Nov 4, 2022 at 12:33 PM Maximilian Michels 
> > >> wrote:
> > >> >
> > >> >> Hey Martijn, hi Chesnay,
> > >> >>
> > >> >> >The big problem of using the ASF Slack instance is that users can
> > join
> > >> >> any Slack channel, including ones outside of Flink.
> > >> >>
> > >> >> That is one of the main motivations for proposing to move to the
> ASF
> > >> >> Slack. We can create an unlimited number of "flink-XY" channels in
> > the
> > >> ASF
> > >> >> Slack, although one or two are probably all we need. It seems
> logical
> > >> that
> > >> >> we share the Slack workspace, just like the other infrastructure
> like
> > >> JIRA,
> > >> >> mail, Jenkins, web server, etc. I guess I'm just in too many Slack
> > >> >> workspaces already.
> > >> >>
> > >> >> From an ASF standpoint, I think it would be desirable to channel
> > people
> > >> >> into the main workspace to promote the software foundation. Also,
> the
> > >> main
> > >> >> point of communication should still be the mailing lists. I wonder
> > how
> > >> that
> > >> >> has played out since the creation of the Slack workspace? I 

Re: [VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0

[Konstantin Knauf]

+1 (binding)

Am Fr., 28. Okt. 2022 um 16:58 Uhr schrieb Piotr Nowojski <
pnowoj...@apache.org>:

> Hi,
>
> As discussed on the dev mailing list [0] I would like to start a vote to
> drop support of older savepoint formats (for Flink versions older than
> 1.8). You can find the original explanation from the aforementioned dev
> mailing list thread at the bottom of this message.
>
> Draft PR containing the proposed change you can find here:
> https://github.com/apache/flink/pull/21056
>
> Vote will be open at least until Wednesday, November 2nd 18:00 CET.
>
> Best,
> Piotrek
>
> [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w
>
> I would like to open a discussion to remove the long deprecated
> (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the related
> code.
>
> The motivation behind this move is two fold. One reason is that it
> complicates our code base unnecessarily and creates confusion on how to
> actually implement custom serializers. The immediate reason is that I
> wanted to clean up Flink's configuration stack a bit and refactor the
> ExecutionConfig class [2]. This refactor would keep the API compatibility
> of the ExecutionConfig, but it would break savepoint compatibility with
> snapshots written with some of the old serializers, which had
> ExecutionConfig as a field and were serialized in the snapshot. This issue
> has been resolved by the introduction of TypeSerializerSnapshot in Flink
> 1.7 [3], where serializers are no longer part of the snapshot.
>
> TypeSerializerConfigSnapshot has been deprecated and no longer used by
> built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged to
> migrate to TypeSerializerSnapshot since then with their own custom
> serializers. That has been plenty of time for the migration.
>
> This proposal would have the following impact for the users:
> 1. we would drop support for recovery from savepoints taken with Flink <
> 1.7.0 for all built in types serializers
> 2. we would drop support for recovery from savepoints taken with Flink <
> 1.8.0 for built in kryo serializers
> 3. we would drop support for recovery from savepoints taken with Flink <
> 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot
>
> 1. and 2. would have a simple migration path. Users migrating from those
> old savepoints would have to first start his job using a Flink version from
> the [1.8, 1.16] range, and take a new savepoint that would be compatible
> with Flink 1.17.
> 3. This is a bit more problematic, because users would have to first
> migrate their own custom serializers to use TypeSerializerSnapshot (using a
> Flink version from the [1.8, 1.16]), take a savepoint, and only then
> migrate to Flink 1.17. However users had already 4 years to migrate, which
> in my opinion has been plenty of time to do so.
>
> As a side effect, we could also drop support for some of the legacy
> metadata serializers from LegacyStateMetaInfoReaders and potentially other
> places that we are keeping for the sake of compatibility with old
> savepoints.
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html
> [2] https://issues.apache.org/jira/browse/FLINK-29379
> [3] https://issues.apache.org/jira/browse/FLINK-9377
> [4] https://issues.apache.org/jira/browse/FLINK-9376
> [5] https://issues.apache.org/jira/browse/FLINK-11323
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Release Flink 1.15.3

[Konstantin Knauf]

+1, thanks Fabian.

Am Mi., 26. Okt. 2022 um 08:26 Uhr schrieb Danny Cranmer <
dannycran...@apache.org>:

> +1, thanks for driving this Fabian.
>
> Danny,
>
> On Wed, Oct 26, 2022 at 2:22 AM yuxia  wrote:
>
> > Thanks for driving this.
> > +1 for release 1.15.3
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "Leonard Xu" 
> > 收件人: "dev" 
> > 发送时间: 星期二, 2022年 10 月 25日 下午 10:00:47
> > 主题: Re: [DISCUSS] Release Flink 1.15.3
> >
> > Thanks Fabian for driving this.
> >
> > +1 to release 1.15.3.
> >
> > The bug tickets FLINK-26394 and FLINK-27148 should be fixed as well, I’ll
> > help to address them soon.
> >
> > Best,
> > Leonard Xu
> >
> >
> >
> > > 2022年10月25日 下午8:28，Jing Ge  写道：
> > >
> > > +1 The timing is good to have 1.15.3 release. Thanks Fabian for
> bringing
> > > this to our attention.
> > >
> > > I just checked PRs and didn't find the 1.15 backport of FLINK-29567
> > > . Please be aware
> of
> > it.
> > > Thanks!
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Tue, Oct 25, 2022 at 11:44 AM Xintong Song 
> > wrote:
> > >
> > >> Thanks for bringing this up, Fabian.
> > >>
> > >> +1 for creating a 1.15.3 release. I've also seen users requiring this
> > >> version [1].
> > >>
> > >> I can help with actions that require a PMC role, if needed.
> > >>
> > >> Best,
> > >>
> > >> Xintong
> > >>
> > >>
> > >> [1] https://lists.apache.org/thread/501q4l1c6gs8hwh433bw3v1y8fs9cw2n
> > >>
> > >>
> > >>
> > >> On Tue, Oct 25, 2022 at 5:11 PM Fabian Paul  wrote:
> > >>
> > >>> Hi all,
> > >>>
> > >>> I want to start the discussion of creating a new 1.15 patch release
> > >>> (1.15.3). The last 1.15 release is almost two months old, and since
> > then,
> > >>> ~60 tickets have been closed, targeting 1.15.3. It includes critical
> > >>> changes to the sink architecture, including:
> > >>>
> > >>> - Reverting the sink metric naming [1]
> > >>> - Recovery problems for sinks using the GlobalCommitter [2][3][4]
> > >>>
> > >>> If the community agrees to create a new patch release, I could
> > volunteer
> > >> as
> > >>> the release manager.
> > >>>
> > >>> Best,
> > >>> Fabian
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/FLINK-29567
> > >>> [2] https://issues.apache.org/jira/browse/FLINK-29509
> > >>> [3] https://issues.apache.org/jira/browse/FLINK-29512
> > >>> [4] https://issues.apache.org/jira/browse/FLINK-29627
> > >>>
> > >>
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Issue tracking workflow

[Konstantin Knauf]

Hi everyone,

while I see some benefits in moving to Github Issues completely, we need to
be aware that Github Issues lacks many features that Jira has. From the top
of my head:
* there are no issue types
* no priorities
* issues can only be assigned to one milestone
So, you need to work a lot with labels and conventions and basically need
bots or actions to manage those. Agreeing on those processes, setting them
up and getting used to them will be a lot of work for the community.

So, I am also in favor of 1) for now, because I don't really see a good
alternative option.

Cheers,

Konstantin



Am Mo., 24. Okt. 2022 um 22:27 Uhr schrieb Matthias Pohl
:

> I agree that leaving everything as is would be the best option. I also tend
> to lean towards option 4 as a fallback for the reasons already mentioned.
> I'm still not a big fan of the Github issues. But that's probably only
> because I'm used to the look-and-feel and the workflows of Jira. I see
> certain benefits of moving to Github, though. We're still having the idea
> of migrating from AzureCI to GitHub Actions. Moving the issues to GitHub as
> well might improve the user experience even more. Reducing the number of
> services a new contributor should be aware of to reach the community is a
> good way to reduce the confusion for newcomers, I could imagine.
> Additionally, I also like the fact that I wouldn't have to bother about the
> Apache Jira markdown anymore. 8)
>
> I agree with Martijn's concern about not being able to track all
> Flink-related issues in a single system. I'm just wondering whether
> something is holding us back from collecting all Flink-related issues in
> the Flink's Github repository and disabling the issue feature in any other
> Flink-related repository?
>
> About migrating the Jira issues: I would be hopeful that migrating is
> doable in the end. There is a blog post from the spring data guys about
> their journey on migrating from Jira to GitHub issues [1]. Unfortunately,
> they didn't provide any scripts.
>
> For the case that infra moves forward with disabling the signup without us
> having come up with a decision and its actual execution (i.e. migrating the
> Jira issues to GH), I would prefer having users send a request to the
> mailing list. I would rather have a temporary phase where there's a bit of
> overhead of registering the users in the Apache Jira than having two
> locations for bug tracking. I suspect that there are no statistics on how
> many new users register with Jira in a given timeframe to contribute to
> Flink?
>
> Matthias
>
> [1]
>
> https://spring.io/blog/2021/01/07/spring-data-s-migration-from-jira-to-github-issues
> [2] https://lists.apache.org/thread/pjb5jzvw41xjtzgf4w0gggpqrt2fq6ov
>
>
> On Mon, Oct 24, 2022 at 10:28 AM Xintong Song 
> wrote:
>
> > I agree with you that option 1) would be the best for us. Let's keep
> hoping
> > for the best.
> >
> > Option 4), as you said, comes with prices. At the moment, I don't have
> > thorough answers to your questions.
> >
> > Just one quick response, I think there's a good chance that we can import
> > current Jira tickets into GH. Jira supports exporting issues with fields
> > that you specified as CSV/XML/... files. With a brief google search, I
> > found some tools that help bulk creating issues in GH. E.g.,
> > github-csv-tools [1] is described to support importing issues with title,
> > body, labels, status and milestones from a CSV file. That's probably good
> > enough for us to search/filter the issues in GH, and a link to the Jira
> > ticket can be posted in the GH issue for complete conversation history
> and
> > other details.
> >
> > Best,
> >
> > Xintong
> >
> >
> > [1] https://github.com/gavinr/github-csv-tools
> >
> >
> >
> > On Mon, Oct 24, 2022 at 3:58 PM Martijn Visser  >
> > wrote:
> >
> > > Hi Xintong,
> > >
> > > I'm also not in favour of option 2, I think that two systems will
> result
> > > in an administrative burden and less-efficient workflow. I'm also not
> in
> > > favour of option 3, I think that this will result in first time
> > > users/contributors in not-filling their first bug report, user question
> > or
> > > feature request.
> > >
> > > I'm still hoping for option 1 while the discussion is not finished with
> > > Infra.
> > >
> > > If we assume that option 1 won't be possible, then I think option 4
> will
> > > be the best-option-left. I'm not necessarily in favour, because of a
> > number
> > > of problems it will introduce:
> > >
> > > 1. I don't think importing current Jira tickets into Github Issues is a
> > > realistic option. So we would have two administrations. Before you
> > create a
> > > new ticket, you should check if it exists both as a Jira ticket and as
> a
> > > Github Issue.
> > > 2. How would we deal with completing a PR? There must be one
> > > administration leading for the changelog generation (to avoid that
> you're
> > > missing an item), which could then only be Github Issues. So would we
> > 

Re: [VOTE] Drop Gelly

[Konstantin Knauf]

+1

Am Do., 13. Okt. 2022 um 10:56 Uhr schrieb Niels Basjes :

> +1
>
> On Wed, Oct 12, 2022 at 11:00 PM Martijn Visser 
> wrote:
>
> > Hi everyone,
> >
> > I would like to open a vote for dropping Gelly, which was discussed a
> long
> > time ago but never put to a vote [1].
> >
> > Voting will be open for at least 72 hours.
> >
> > Best regards,
> >
> > Martijn
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
> > [1] https://lists.apache.org/thread/2m6wtgjvxcogbf9d5q7mqt4ofqjf2ojc
> >
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS][FLINK-29372] Add a suffix to keys that violate YAML spec

[Konstantin Knauf]

Make sense to me. It is moving us in the right direction and makes it
possible to drop these keys with Flink 2.0 if that ever happens :)

Am Mi., 21. Sept. 2022 um 16:06 Uhr schrieb Chesnay Schepler <
ches...@apache.org>:

> Hi,
>
> we have a small number of options in Flink whose key is a prefix of
> other keys.
>
> This violates the YAML spec; when you view the options as a tree only
> leaf nodes may have properties.
>
> While this is a minor issue from our side I think this can be quite
> annoying for users, since it means you can't read or validate a Flink
> config with standard yaml tools.
>
> I'd like to add a suffix to those keys to resolve this particular
> problem, while still supporting the previous keys (as deprecated).
>
> AFAICT there aren't any risks here,
> except if users have a search step for one of these options in
> the default config of the Flink distribution;
> however this seems unsafe in any case since the contents of the default
> config may change.
>
> This would also bring us a step closer to our goal of using a compliant
> YAML parser.
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Stale PR action for flink and flink-kubernetes-operator repos

[Konstantin Knauf]

Hi everyone,

thanks for starting this discussion. I think, the flink-kubernetes and
flink repository are too different (age, size) to draw conclusions from
another. On a first thought, I am not a fan of having different processes
for different repositories inside the project.

The discussion reminds me of the discussion on the flink-jira-bot, which
does something similar as suggested for PRs, but on Jira tickets and with
more relaxed rules.

I am personally unsure what's the best way forward. On the one hand, 1000
Open PRs is daunting both for reviewers and people who open new PRs +
realistically 90% or so will never be merged at this point, because they
are too old and stale. So, letting a bot clean this up, would be honest and
could get us into a situation where Committers are motivated to try to keep
the number low again. On the other hand, closing valid PRs after 60 days
just because no one had time to review it might be unnecessarily
harsh. @Chesnay
Schepler  What were other reasons for not doing this in
the past?

Cheers,

Konstantin





Am Mi., 14. Sept. 2022 um 16:35 Uhr schrieb Márton Balassi <
balassi.mar...@gmail.com>:

> Hi gents,
>
> Thanks for the stats Martjin, that valuable insight into the situation.
> Having a large number of open, stale PRs can also result in a bad
> contributor experience down the line, as in my opinion it can reach a point
> where it discourages committers reviewing them as it starts to feel like an
> effort too large to tackle.
>
> Would you be comfortable if we experimented with the approach G is
> suggesting on the kubernetes operator repo to gather some experience and
> shared it here?
>
> Cheers,
> Marton
>
> On Wed, Sep 14, 2022 at 3:42 PM Chesnay Schepler 
> wrote:
>
> > We've had discussions about closing stale PRs several times in the past
> > and always rejected it.
> >
> > I see no reason to change this.
> > If you want to close a PR, then do so while informing the contributor
> > about the reason.
> >
> > On 14/09/2022 15:36, Martijn Visser wrote:
> > > Hi Gabor,
> > >
> > > I have my doubts: I think the majority of the open PRs are not open
> > because
> > > of inactivity from the contributor, but I think the majority (at least
> > for
> > > the flink repository) are open because there are not enough reviews
> > > occurring. If we actively mark those as stale and close them, I think
> it
> > > makes a bad impression towards the contributor, since they can't
> > influence
> > > getting a review in.
> > >
> > > Some numbers for the Flink repo: there's now 957 open PRs, of which 839
> > > haven't been reviewed. That's 88%. 91 PRs have been reviewed and
> > > changes have been requested, which is almost 10%.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > Op wo 14 sep. 2022 om 15:30 schreef Őrhidi Mátyás <
> > matyas.orh...@gmail.com>:
> > >
> > >> Hi Gabor,
> > >>
> > >> Thanks for bringing this to our attention. I'd be happy to see such
> > >> automatism guarding our repos. We could start a trial period on the
> > >> operator repo I guess until we have the confidence it's a good thing.
> > Are
> > >> you aware of this plugin being used at other ASF projects? Any
> > pros/cons?
> > >>
> > >> Cheers,
> > >> Matyas
> > >>
> > >>
> > >> On Wed, Sep 14, 2022 at 2:58 PM Gabor Somogyi <
> > gabor.g.somo...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi All,
> > >>>
> > >>> As I see there is no action for stale PRs for flink and
> > >>> flink-kubernetes-operator repos however almost 1k PRs are open.
> > >>>
> > >>> I would like to suggest to add new stale PR action based on the
> > following
> > >>> github plugin:
> > >>> https://github.com/marketplace/actions/close-stale-issues
> > >>>
> > >>> I think the default values for the plugin looks sufficient to our
> > needs:
> > >>> * Add a label "Stale" on pull requests after 60 days of inactivity
> and
> > >>> comment on them
> > >>> * Close the stale pull requests after 7 days of inactivity
> > >>> * If an update/comment occur on stale issues or pull requests, the
> > stale
> > >>> label will be removed and the timer will restart
> > >>>
> > >>> A playground repo is created to test the feature here:
> > >>> https://github.com/gaborgsomogyi/stale-test-repo
> > >>>
> > >>> Waiting on opinions/suggestions to make our PR queue manageable.
> > >>>
> > >>> Thanks in advance!
> > >>>
> > >>> BR,
> > >>> G
> > >>>
> >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Releasing Flink 1.14.6

[Konstantin Knauf]

Sounds good. +1.

Am Di., 6. Sept. 2022 um 10:45 Uhr schrieb Jingsong Li <
jingsongl...@gmail.com>:

> +1 for 1.14.6
>
> Thanks Xingbo for driving.
>
> Best,
> Jingsong
>
> On Tue, Sep 6, 2022 at 4:42 PM Xingbo Huang  wrote:
> >
> > Hi all,
> >
> > I would like to start discussing releasing Flink 1.14.6.
> >
> > It has already been almost three months since we released 1.14.5. There
> are
> > currently 35 tickets[1] and 33 commits[2] already resolved for 1.14.6,
> some
> > of them quite important, such as FLINK-27399
> >  and FLINK-29138
> > .
> >
> > Currently, there are no issues marked as critical or blocker for 1.14.6.
> > Please let me know if there are any issues you'd like to be included in
> > this release but still not merged.
> >
> > I would like to volunteer as a release manager for 1.14.6, and start the
> > release process once all the issues are merged.
> >
> > Best,
> > Xingbo
> >
> > [1]
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%201.14.6
> > [2]
> https://github.com/apache/flink/compare/release-1.14.5...release-1.14
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE] FLIP-258 Guarantee binary compatibility for Public/-Evolving APIs between patch releases​

[Konstantin Knauf]

+1. Thanks, Chesnay.

Am Di., 6. Sept. 2022 um 11:51 Uhr schrieb Chesnay Schepler <
ches...@apache.org>:

> Since no one objected in the discuss thread, let's vote!
>
> FLIP:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152857
>
> The vote will be open for at least 72h.
>
> Regards,
> Chesnay
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-254 Redis Streams connector

[Konstantin Knauf]

Hi Martijn, Hi Mohammad,

thanks for creating this FLIP.

I am wondering, do you know how much Redis Streams are used compared to
other Redis data structures? At least as a sink and LookupTableSource
supporting regular Redis data structures seems quite useful.

I suppose such connectors would live in the same repository. Would be good
to consider when naming the Redis Streams connector.

Cheers,

Konstantin



Am Mi., 31. Aug. 2022 um 13:55 Uhr schrieb Martijn Visser <
martijnvis...@apache.org>:

> Hi everyone,
>
> Last call for comments, else I'll open up a VOTE thread later this week.
>
> Best regards,
>
> Martijn
>
> Op ma 1 aug. 2022 om 15:40 schreef Martijn Visser <
> martijnvis...@apache.org
> >:
>
> > Hi everyone,
> >
> > Together with Mohammad we would like to start a discussion thread on
> > FLIP-254 for a Redis Streams connector. This would also be a new
> connector
> > in its own external connector repository, being
> > https://www.github.com/apache/flink-connector-redis. This repository
> > already exists, because there was an open PR to add this to the Flink
> main
> > repository and they were planned to be one of the first movers to the
> > external connector repo. However, we never formally agreed that we want
> > this, therefore we're now creating this FLIP.
> >
> > Looking forward to any comments or feedback. If not, I'll open a VOTE
> > thread in the next couple of days.
> >
> > Best regards,
> >
> > Martijn
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-254 Guarantee binary compatibility for Public/-Evolving APIs between patch releases

[Konstantin Knauf]

Hi Chesnay,

thanks for bringing this up and for your research and fixes to japicmd.

+1 for the proposal. For Immerok as an Apache Flink cloud service provider
it is very valuable to know that our users don't need to upgrade their Jobs
when the Flink patch version changes. I am sure the same is true for
internal platform teams as well as end users of Apache Flink.

Cheers,

Konstantin





Am Mi., 31. Aug. 2022 um 12:31 Uhr schrieb Chesnay Schepler <
ches...@apache.org>:

> Hello,
>
> I just published a FLIP to guarantee binary compatibility for patch
> releases. I don't think our current guarantees of source-compatibility
> are sufficient for patch releases.
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152857
>
> Let me know what you think.
>
> Regards,
> Chesnay
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: Rescuing Queryable State from deprecation

[Konstantin Knauf]

Hi Ron,

thanks you for sharing your use case and your initiative to save Queryable
State. Queryable State was deprecated due to a lack of maintainers and thus
the community did not have resources to improve and develop it further. The
deprecation notice was added to signal this lack of attention to our
users.  On the other hand, I am not aware of anyone who is actively working
towards actually dropping Queryable State. So, I don't think this will
happen anytime soon.

Personally, I see a lot of potential in Queryable State, but to make it
really, really useful it still needs a lot of work and the current
architecture/implementation might need to be re-thought from scratch in my
opinion. In that case, the current implementation might need to phased out
again, but opposed to today there would be an alternative then. I am not
aware of anyone at Immerok who currently has time to review & support a
major effort in this direction. Just keeping the current implementation
alive without any fundamental changes might not be too much work, though.

So, all in all, I think, Queryable State will neither be fundamentally
improved nor dropped any time soon. If you are committed to maintaining the
current implementation of Queryable State in terms of addressing bugs, test
instabilities and potentially small features requests, I am +1 to remove it
from the deprecation list again provided that we keep the
APIs @Experimental.

Cheers,

Konstantin







Am Mo., 29. Aug. 2022 um 01:30 Uhr schrieb Ron Crocker
:

> Hi -
>
> For those of you who didn’t see my talk from Flink Forward, here’s a link <
> https://www.slideshare.net/FlinkForward/using-queryable-state-for-fun-and-profit>
> to the slides.
> By implementing the ideas first brought forward in the blog post <
> https://www.ververica.com/blog/queryable-state-use-case-demo>, we’ve
> found we can save a pretty substantial amount of expense using Flink
> Queryable State as a replacement for certain off-board caches.
>
> Given that, I’m quite keen on rescuing queryable state from deprecation.
> However, I’m not at all sure how to start that conversation (other than by
> just starting it).
>
> I believe my argument for rescuing this feature is supported in my talk,
> though I’m open to further discussion.
>
> Perhaps the next thing to understanding is why, in concrete terms,
> queryable state was added to the deprecation list. Can someone enlighten me
> on that?
>
> Ron
> —
> Ron Crocker
> New Relic Fellow & Architect
> ( ( •)) New Relic
> rcroc...@newrelic.com
> M: +1 630 363 8835
>
>

-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE] Release 1.15.2, release candidate #2

[Konstantin Knauf]

+1 (binding)

* checked checksum of binaries
* checked signatures of binaries & Maven artifacts
* checked dependency & NOTICE changes
* ran TopSpeedWindowing locally

Thanks for driving the release, Danny! Good job.


Am Di., 23. Aug. 2022 um 04:16 Uhr schrieb Peng Kang
:

> +1
>
> -- Forwarded message -
> From: Dawid Wysakowicz 
> Date: Mon, Aug 22, 2022 at 7:21 PM
> Subject: Re: [VOTE] Release 1.15.2, release candidate #2
> To: 
> Cc: Danny Cranmer 
>
>
> +1 (binding)
>
> - signatures & checksums OK
> - checked changed licenses from 1.15.1
> - PR OK
> - no excessive or binary files in the source distribution
>
> Best,
>
> Dawid
>
> On 19.08.2022 10:30, Xingbo Huang wrote:
> > +1 (non-binding)
> >
> > - verify signatures and checksums
> > - no binaries found in source archive
> > - reviewed the release note blog
> > - verify python wheel package contents
> > - pip install apache-flink-libraries and apache-flink wheel packages
> > - run the examples from Python Table API tutorial
> >
> > Best,
> > Xingbo
> >
> > Chesnay Schepler  于2022年8月19日周五 15:51写道：
> >
> >> +1 (binding)
> >>
> >> - signatures OK
> >> - all required artifacts on dist.apache.org
> >> - maven artifacts appear complete
> >> - tag exists
> >> - PR OK
> >> - no PaxHeader directories
> >> - no excessive files in the distribution
> >>
> >> On 17/08/2022 19:52, Danny Cranmer wrote:
> >>> Hi everyone,
> >>>
> >>> Please review and vote on the release candidate #2 for the version
> >> 1.15.2,
> >>> as follows:
> >>> [ ] +1, Approve the release
> >>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>
> >>> The complete staging area is available for your review, which includes:
> >>>
> >>>  - JIRA release notes [1],
> >>>  - the official Apache source release and binary convenience
> releases
> >> to
> >>>  be deployed to dist.apache.org [2], which are signed with the key
> >> with
> >>>  fingerprint 125FD8DB [3],
> >>>  - all artifacts to be deployed to the Maven Central Repository
> [4],
> >>>  - source code tag "release-1.15.2-rc2" [5],
> >>>  - website pull request listing the new release and adding
> >> announcement
> >>>  blog post [6].
> >>>
> >>>
> >>> The vote will be open for at least 72 hours. It is adopted by majority
> >>> approval, with at least 3 PMC affirmative votes.
> >>>
> >>> 1.15.2-RC1 was rejected for 2x issues:
> >>>
> >>>  1. Dist/src archives contained PaxHeader files when decompressing
> on
> >>>  Windows. Root cause was tar default archive format on Mac, fixed
> by
> >> using
> >>>  gnu-tar. I will follow up to update the release process to avoid
> >> this issue
> >>>  in the future.
> >>>  2. Dist/src archives contained additional files. I had some
> locally
> >>>  gitignored files that ended up in the archive. New build used a
> >> fresh clone
> >>>  of Flink and I compared the archive contents of 1.15.1 with
> 1.15.2.
> >>>
> >>>
> >>> Thanks,
> >>> Danny Cranmer
> >>>
> >>> [1]
> >>>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351829
> >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.15.2-rc2/
> >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >>> [4]
> >> https://repository.apache.org/content/repositories/orgapacheflink-1524
> >>> [5] https://github.com/apache/flink/tree/release-1.15.2-rc2
> >>> [6] https://github.com/apache/flink-web/pull/566
> >>>
> >>
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Releasing Flink 1.15.2

[Konstantin Knauf]

Hi Guillaume,

as long as those vulnerabilities have not been introduced since Flink
1.15.1, which they haven't, we can proceed with should not block the
release.

Best,

Konstantin

Am Di., 23. Aug. 2022 um 10:12 Uhr schrieb Guillaume Vauvert <
guillaume.vauvert@gmail.com>:

> Hello all,
>
> I am wondering if it is safe to release 1.15.2 despite
> https://issues.apache.org/jira/browse/FLINK-29065 ?
>
> Regards,
>
> Guillaume
>
> On 8/16/22 12:53, Danny Cranmer wrote:
> > Hello all,
> >
> > I have created a PR [1] to update the documentation to cover the 1.15.x
> > stateful upgrades. Would appreciate a review for correctness. I will also
> > put a note in the release announcement blog post.
> >
> > We also have another blocker [2] opened against 1.15.2. I will proceed
> with
> > the release preparation but am expecting this one to need to go in.
> >
> > [1] https://github.com/apache/flink/pull/20600
> > [2] https://issues.apache.org/jira/browse/FLINK-28975
> >
> > Thanks,
> > Danny
> >
> > On Mon, Aug 15, 2022 at 2:02 PM Timo Walther  wrote:
> >
> >> Thanks Danny.
> >>
> >> I will merge FLINK-28861 in couple of minutes to master. I will open a
> >> PR for 1.15 shortly. This issue is pretty tricky, we should add a
> >> warning to 1.15.0 and 1.15.1 releases as it won't be easy to perform
> >> stateful upgrades in between 1.15.x patch versions for pipelines that
> >> use Table API.
> >>
> >> Regards,
> >> Timo
> >>
> >> On 15.08.22 10:42, Danny Cranmer wrote:
> >>> Thanks all.
> >>>
> >>> I can see the final issue has an approved PR now, awaiting merge,
> thanks
> >>> Timo and Chesnay. In the meantime I will get setup ready to start the
> >>> release process.
> >>>
> >>> Thanks,
> >>>
> >>> On Fri, Aug 12, 2022 at 10:47 PM Jing Ge  wrote:
> >>>
>  Thanks Danny! Strong +1 and looking forward to the 1.15.2 asap.
> 
>  Best regards,
>  Jing
> 
>  On Fri, Aug 12, 2022 at 4:25 AM Xingbo Huang 
> >> wrote:
> > Hi Danny,
> >
> > Thanks for driving the release. +1 for the 1.15.2 release.
> >
> > Best,
> > Xingbo
> >
> > Chesnay Schepler  于2022年8月11日周四 20:06写道：
> >
> >> I think that's a good idea; a user in the Flink slack was asking for
> >> it
> >> just yesterday.
> >>
> >> About FLINK-28861, let's wait a day or something because there
> should
>  be
> >> PR very soon.
> >>
> >> It's perfectly fine for you to be the release manager; I can help
> you
> >> out with things with that require PMC permissions.
> >>
> >> On 11/08/2022 12:39, Danny Cranmer wrote:
> >>> Hello all,
> >>>
> >>> I would like to start discussing the release of Flink 1.15.2. Flink
> >> 1.15.1
> >>> was released on 6th July [1] and we have resolved 31 issues since
>  then
> >> [2].
> >>> During the Flink 1.15.1 vote [3] we identified one blocker [4] and
>  one
> >>> critical issue [5] that should be fixed soon after. We also have 3
> > other
> >>> non-test related critical fixes merged into 1.15.2.
> >>>
> >>> There is 1 blocker in 1.15.2 not yet resolved that we could
> consider
> >>> waiting for, "Cannot resume from savepoint when using
> > fromChangelogStream
> >>> in upsert mode" [6]. There have been discussions on this issue and
> it
> >> looks
> >>> like Timo will be working on it, however there is no PR available
>  yet.
> >>> I'd like to advocate this release and in doing so nominate myself
> to
>  be
> >> the
> >>> release manager. I'm conscious that I have not performed this duty
> >> before,
> >>> so alternatively am happy to shadow this process if I can find a
> > willing
> >>> volunteer to cover it on this occasion.
> >>>
> >>> [1] https://flink.apache.org/news/2022/07/06/release-1.15.1.html
> >>> [2]
> >>>
> >>
> https://issues.apache.org/jira/browse/FLINK-28322?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.15.2%20AND%20status%20in%20(resolved%2C%20closed)%20order%20by%20priority%20desc
> >>> [3]
> https://lists.apache.org/thread/t27qc36g141kzk8d83jytkdshfpdj0xl
> >>> [4] https://issues.apache.org/jira/browse/FLINK-28322
> >>> [5] https://issues.apache.org/jira/browse/FLINK-23528
> >>> [6] https://issues.apache.org/jira/browse/FLINK-28861
> >>>
> >>> Thanks,
> >>> Danny Cranmer
> >>>
> >>
> >>
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE] FLIP-252: Amazon DynamoDB Sink Connector

[Konstantin Knauf]

+1. Thanks!

Am Mi., 20. Juli 2022 um 16:48 Uhr schrieb Tzu-Li (Gordon) Tai <
tzuli...@apache.org>:

> +1
>
> On Wed, Jul 20, 2022 at 6:13 AM Danny Cranmer 
> wrote:
>
> > Hi there,
> >
> > After the discussion in [1], I’d like to open a voting thread for
> FLIP-252
> > [2], which proposes the addition of an Amazon DynamoDB sink based on the
> > Async Sink [3].
> >
> > The vote will be open until July 23rd earliest (72h), unless there are
> any
> > binding vetos.
> >
> > Cheers, Danny
> >
> > [1] https://lists.apache.org/thread/ssmf2c86n3xyd5qqmcdft22sqn4qw8mw
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-252%3A+Amazon+DynamoDB+Sink+Connector
> > [3]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE] FLIP-251: Support collecting arbitrary number of streams

[Konstantin Knauf]

+1 (binding)

Am Mi., 20. Juli 2022 um 15:48 Uhr schrieb David Anderson <
dander...@apache.org>:

> +1
>
> Thank you Chesnay.
>
> On Tue, Jul 19, 2022 at 3:09 PM Alexander Fedulov  >
> wrote:
>
> > +1
> > Looking forward to using the API to simplify tests setups.
> >
> > Best,
> > Alexander Fedulov
> >
> > On Tue, Jul 19, 2022 at 2:31 PM Martijn Visser  >
> > wrote:
> >
> > > Thanks for creating the FLIP and opening the vote Chesnay.
> > >
> > > +1 (binding)
> > >
> > > Op di 19 jul. 2022 om 10:26 schreef Chesnay Schepler <
> ches...@apache.org
> > >:
> > >
> > > > I'd like to proceed with the vote for FLIP-251 [1], as no objections
> or
> > > > issues were raised in [2].
> > > >
> > > > The vote will last for at least 72 hours unless there is an objection
> > or
> > > > insufficient votes.
> > > >
> > > >   [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-251%3A+Support+collecting+arbitrary+number+of+streams
> > > >   [2]
> https://lists.apache.org/thread/ksv71m7rvcwslonw07h2qnw77zpqozvh
> > > >
> > > >
> > >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: Usage of PublicEvolving annotation

[Konstantin Knauf]

Hi Blake,

thank you for reaching out. @PublicEvolving interface can only be changed
with a minor release of Apache Flink (e.g. 1.6). While the JavaDoc is
indeed ambiguous there was a FLIP recently that discussed this topic:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-196%3A+Source+API+stability+guarantees

If you have time, feel free to open a PR to align the JavaDocs. Happy to
merge that.

Cheers,

Konstantin

Am So., 10. Juli 2022 um 07:07 Uhr schrieb Blake Wilson <
bl...@yellowpapersun.com>:

> Hello,
> I've noticed the usage of "PublicEvolving" in some Flink classes described
> as follows
>
> /**
>  * Annotation to mark classes and methods for public use, but with evolving
> interfaces.
>  *
>  * Classes and methods with this annotation are intended for public use
> and have stable behavior.
>  * However, their interfaces and signatures are not considered to be stable
> and might be changed
>  * across versions.
>  *
>  * This annotation also excludes methods and classes with evolving
> interfaces / signatures within
>  * classes annotated with {@link Public}.
>  */
>
> This seems to indicate that these classes can be changed even within a
> patch version of Flink, though the statement "might be changed across
> versions" is a bit ambiguous in this regard.
>
> Is my statement correct? I'm wondering how important it is to preserve the
> API in classes annotated like this. For example, would it be considered
> appropriate to modify an interface if not doing so would require
> introducing a second version of the interface to support a new feature?
>
> This relates specifically to this change
> https://github.com/apache/flink/pull/19417
>
> Thank you
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-251: Support collecting arbitrary number of streams

[Konstantin Knauf]

Makes sense to me. Thank you, Chesnay. +1

David Anderson  schrieb am Fr., 8. Juli 2022, 13:56:

> I've found that with our current tooling it's frustrating to try to write
> good end-to-end tests for real-world jobs with multiple sinks.
> DataStream#executeAndCollect() is okay for simple pipelines with one sink,
> but in my opinion we do need something like FLIP-251.
>
> The proposed interface looks good to me. I look forward to trying it.
>
> Chesnay, thanks for putting this together!
>
> David
>
> On Thu, Jul 7, 2022 at 7:35 AM Chesnay Schepler 
> wrote:
>
> > Hello,
> >
> > I have created a FLIP for slightly extending the collect() API to
> > support collecting an arbitrary number of streams, to make it more
> > useful for testing complex workflows.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-251%3A+Support+collecting+arbitrary+number+of+streams
> >
> > Looking forward to any feedback.
> >
> > Regards,
> >
> > Chesnay
> >
> >
>

Re: [VOTE] Release 1.15.1, release candidate #1

[Konstantin Knauf]

Thanks Jingsong. I agree that this is a serious bug and we need to take
compatibility seriously. We shouldn't wait long until we release Flink
1.15.2. Still I stand to my point, that we shouldn't abort the current
release.

---

+1 (binding)

* checked release blog post
* verified signatures and checksums
* all artifacts present

Best,

Konstantin






Am Mo., 4. Juli 2022 um 03:46 Uhr schrieb Jingsong Li <
jingsongl...@gmail.com>:

> Hi Konstantin,
>
> I have no problem with canceling -1.
>
> However, 1.15.1 will also be a version that is not compatible with 1.14.
> I've heard too many complaints about connector compatibility, and
> Flink always introduces incompatible changes to the connector in every
> release.
> Flink's new version can let connector developers modify code, but
> incompatibilities like FLINK-28322 cause connectors to have to split
> their code, one for 1.15 and one for 1.14, making the connector side
> difficult to maintain.
>
> > Regressions - in this case between 1.15.0 and 1.15.1 - would usually
> justify aborting the release candidate
>
> My understanding is that as long as there are high impact bugs or
> compatibility issues, they are blockers, not just the regressions
> brought about in the new fixes for 1.15.1.
>
> Best,
> Jingsong
>
> On Sat, Jul 2, 2022 at 2:09 AM Konstantin Knauf  wrote:
> >
> > Hi everyone,
> >
> > I would like to raise the question if we should abort this release
> > candidate at the current stage because of newly found bugs?
> >
> > * The minimal voting period had already passed and with one addition
> > binding vote (e.g. mine), we could release this immediately and ship all
> > the other fixes to users.
> >
> > * The way I see the process at this stage of the release, we have already
> > decided that we want to release now (otherwise we wouldn't have prepared
> a
> > candidate) and this is only about sanity checking and validating the
> > release candidate. Regressions - in this case between 1.15.0 and 1.15.1 -
> > would usually justify aborting the release candidate, but as far as I can
> > see neither FLINK-23528 and FLINK-28322 are regressions.
> >
> > @Danny: I don't understand how FINK-23528 is a blocker for this release
> > candidate, sorry. It's been open for almost a year. Of course, it should
> > then go into the next patch release of Flink 1.15.
> >
> > @Jingsong: Please re-consider your -1 vote.
> >
> > Thanks,
> >
> > Konstantin
> >
> >
> > Am Do., 30. Juni 2022 um 18:21 Uhr schrieb Danny Cranmer <
> > dannycran...@apache.org>:
> >
> > > Hello all,
> > >
> > > -1
> > >
> > > The Kinesis Data Streams consumer does not work with Stop With
> Savepoint
> > > [1]. We are planning to have a fix ready to merge tomorrow and would
> > > appreciate getting this in 1.15.1.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-23528
> > >
> > > Thanks,
> > > Danny
> > >
> > > On Thu, Jun 30, 2022 at 9:31 AM Jingsong Li 
> > > wrote:
> > >
> > > > Hi David, Thanks for creating this RC.
> > > >
> > > > -1
> > > >
> > > > We found an incompatible modification in 1.15.0 [1]
> > > > I think we should fix it.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-28322
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > On Tue, Jun 28, 2022 at 8:45 PM Robert Metzger 
> > > > wrote:
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > - staging repo contents look fine
> > > > > - KEYS file ok
> > > > > - binaries start locally properly. WebUI accessible on Mac.
> > > > >
> > > > > On Mon, Jun 27, 2022 at 11:21 AM Qingsheng Ren 
> > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > - checked/verified signatures and hashes
> > > > > > - checked that all POM files point to the same version
> > > > > > - built from source, without Hadoop and using Scala 2.12
> > > > > > - started standalone cluster locally, WebUI is accessiable and
> ran
> > > > > > WordCount example successfully
> > > > > > - executed a job with SQL client consuming from Kafka source to
> > > collect
> > > > > > sink
> > > > > >
> > > >

Re: [VOTE] Release 1.15.1, release candidate #1

[Konstantin Knauf]

Hi everyone,

I would like to raise the question if we should abort this release
candidate at the current stage because of newly found bugs?

* The minimal voting period had already passed and with one addition
binding vote (e.g. mine), we could release this immediately and ship all
the other fixes to users.

* The way I see the process at this stage of the release, we have already
decided that we want to release now (otherwise we wouldn't have prepared a
candidate) and this is only about sanity checking and validating the
release candidate. Regressions - in this case between 1.15.0 and 1.15.1 -
would usually justify aborting the release candidate, but as far as I can
see neither FLINK-23528 and FLINK-28322 are regressions.

@Danny: I don't understand how FINK-23528 is a blocker for this release
candidate, sorry. It's been open for almost a year. Of course, it should
then go into the next patch release of Flink 1.15.

@Jingsong: Please re-consider your -1 vote.

Thanks,

Konstantin


Am Do., 30. Juni 2022 um 18:21 Uhr schrieb Danny Cranmer <
dannycran...@apache.org>:

> Hello all,
>
> -1
>
> The Kinesis Data Streams consumer does not work with Stop With Savepoint
> [1]. We are planning to have a fix ready to merge tomorrow and would
> appreciate getting this in 1.15.1.
>
> [1] https://issues.apache.org/jira/browse/FLINK-23528
>
> Thanks,
> Danny
>
> On Thu, Jun 30, 2022 at 9:31 AM Jingsong Li 
> wrote:
>
> > Hi David, Thanks for creating this RC.
> >
> > -1
> >
> > We found an incompatible modification in 1.15.0 [1]
> > I think we should fix it.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-28322
> >
> > Best,
> > Jingsong
> >
> > On Tue, Jun 28, 2022 at 8:45 PM Robert Metzger 
> > wrote:
> > >
> > > +1 (binding)
> > >
> > > - staging repo contents look fine
> > > - KEYS file ok
> > > - binaries start locally properly. WebUI accessible on Mac.
> > >
> > > On Mon, Jun 27, 2022 at 11:21 AM Qingsheng Ren 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > - checked/verified signatures and hashes
> > > > - checked that all POM files point to the same version
> > > > - built from source, without Hadoop and using Scala 2.12
> > > > - started standalone cluster locally, WebUI is accessiable and ran
> > > > WordCount example successfully
> > > > - executed a job with SQL client consuming from Kafka source to
> collect
> > > > sink
> > > >
> > > > Best,
> > > > Qingsheng
> > > >
> > > >
> > > > > On Jun 27, 2022, at 14:46, Xingbo Huang 
> wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > - verify signatures and checksums
> > > > > - no binaries found in source archive
> > > > > - build from source
> > > > > - Reviewed the release note blog
> > > > > - verify python wheel package contents
> > > > > - pip install apache-flink-libraries and apache-flink wheel
> packages
> > > > > - run the examples from Python Table API tutorial
> > > > >
> > > > > Best,
> > > > > Xingbo
> > > > >
> > > > > Chesnay Schepler  于2022年6月24日周五 21:42写道：
> > > > >
> > > > >> +1 (binding)
> > > > >>
> > > > >> - signatures OK
> > > > >> - all required artifacts appear to be present
> > > > >> - tag exists with the correct version adjustments
> > > > >> - binary shows correct commit and version
> > > > >> - examples run fine
> > > > >> - website PR looks good
> > > > >>
> > > > >> On 22/06/2022 14:20, David Anderson wrote:
> > > > >>> Hi everyone,
> > > > >>>
> > > > >>> Please review and vote on release candidate #1 for version
> 1.15.1,
> > as
> > > > >>> follows:
> > > > >>> [ ] +1, Approve the release
> > > > >>> [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > >>>
> > > > >>> The complete staging area is available for your review, which
> > includes:
> > > > >>>
> > > > >>> * JIRA release notes [1],
> > > > >>> * the official Apache source release and binary convenience
> > releases to
> > > > >> be
> > > > >>> deployed to dist.apache.org [2], which are signed with the key
> > with
> > > > >>> fingerprint E982F098 [3],
> > > > >>> * all artifacts to be deployed to the Maven Central Repository
> [4],
> > > > >>> * source code tag "release-1.15.1-rc1" [5],
> > > > >>> * website pull request listing the new release and adding
> > announcement
> > > > >> blog
> > > > >>> post [6].
> > > > >>>
> > > > >>> The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > >>> approval, with at least 3 PMC affirmative votes.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> David
> > > > >>>
> > > > >>> [1]
> > > > >>>
> > > > >>
> > > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=
> > > > >>> 12351546
> > > > >>> [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-1.15.1-rc1/
> > > > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > >>> [4]
> > > > >>
> > https://repository.apache.org/content/repositories/orgapacheflink-1511/
> > > > >>> [5] https://github.com/apache/flink/tree/release-1.15.1-rc1
> > > > >>> [6] 

Re: [DISCUSS] Releasing 1.15.1

[Konstantin Knauf]

Sounds like a good plan. Thank you!

Am Fr., 17. Juni 2022 um 12:33 Uhr schrieb David Anderson <
dander...@apache.org>:

> After hitting pause on the 1.15.1 release a couple of days ago because of
> FLINK-28060 [1], I want to hit resume now. You should go read that ticket
> if you want the details, but the summary is that the upgrade of the Kafka
> Clients to 2.8.1 that was done in Flink 1.15.0 is causing a bug that means
> that after a Kafka broker restarts, subsequent attempts by Flink to commit
> Kafka offsets will fail until Flink is restarted. The only good fix
> available to us appears to be upgrading the Kafka Clients to version 3.1.1.
>
> It doesn't seem wise to upgrade the Kafka Clients from 2.8.1 to 3.1.1 for
> Flink 1.15.1. That's a big change to make this close to our release, and we
> shouldn't delay this release any further. So here's a proposal:
>
>- We bump the Kafka Clients to 3.1.1 in master now.
>- We don't try to fix FLINK-28060 for 1.15.1.
>
>
>- We create the Flink 1.15.1 release straight away, noting that there's
>a known issue with Kafka (FLINK-28060).
>- We reach out to the Kafka community to see if they're willing to
>create a 2.8.2 release with a backport of the fix for this bug.
>- In parallel, we merge a bump of the Kafka Clients to 3.1.1 after
>1.15.1 is done, to see how it behaves on the CI for the next few weeks,
> and
>plan a quick Flink 1.15.2 release (most likely something like a month
>later).
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-28060
>
> Best,
> David
>
> On Wed, Jun 15, 2022 at 11:37 AM David Anderson 
> wrote:
>
> > I'm now thinking we should delay 1.15.1 long enough to see if we can
> > include a fix for FLINK-28060 [1], which is a serious regression
> affecting
> > several Kafka users.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-28060
> >
> > On Fri, Jun 10, 2022 at 12:15 PM David Anderson 
> > wrote:
> >
> >> Since no one has brought up any blockers, I'll plan to start the release
> >> process on Monday unless I hear otherwise.
> >>
> >> Best,
> >> David
> >>
> >> On Thu, Jun 9, 2022 at 10:20 AM Yun Gao 
> >> wrote:
> >>
> >>> Hi David,
> >>>
> >>> Very thanks for driving the new version, also +1 since we already
> >>> accumulated some fixes.
> >>>
> >>> Regarding https://issues.apache.org/jira/browse/FLINK-27492, currently
> >>> there are still some
> >>> controversy with how to deal with the artifacts. I also agree we may
> not
> >>> hold up the release
> >>> with this issue. We'll try to reach to the consensus as soon as
> possible
> >>> to try best catching
> >>> up with the release.
> >>>
> >>> Best,
> >>> Yun
> >>>
> >>>
> >>>
> >>> --
> >>> From:LuNing Wang 
> >>> Send Time:2022 Jun. 9 (Thu.) 10:10
> >>> To:dev 
> >>> Subject:Re: [DISCUSS] Releasing 1.15.1
> >>>
> >>> Hi David,
> >>>
> >>> +1
> >>> Thank you for driving this.
> >>>
> >>> Best regards,
> >>> LuNing Wang
> >>>
> >>> Jing Ge  于2022年6月8日周三 20:45写道：
> >>>
> >>> > +1
> >>> >
> >>> > Thanks David for driving it!
> >>> >
> >>> > Best regards,
> >>> > Jing
> >>> >
> >>> >
> >>> > On Wed, Jun 8, 2022 at 2:32 PM Xingbo Huang 
> >>> wrote:
> >>> >
> >>> > > Hi David,
> >>> > >
> >>> > > +1
> >>> > > Thank you for driving this.
> >>> > >
> >>> > > Best,
> >>> > > Xingbo
> >>> > >
> >>> > > Chesnay Schepler  于2022年6月8日周三 18:41写道：
> >>> > >
> >>> > > > +1
> >>> > > >
> >>> > > > Thank you for proposing this. I can take care of the PMC-side of
> >>> > things.
> >>> > > >
> >>> > > > On 08/06/2022 12:37, Jingsong Li wrote:
> >>> > > > > +1
> >>> > > > >
> >>> > > > > Thanks David for volunteering to manage the release.
> >>> > > > >
> >>> > > > > Best,
> >>> > > > > Jingsong
> >>> > > > >
> >>> > > > > On Wed, Jun 8, 2022 at 6:21 PM Jark Wu 
> wrote:
> >>> > > > >> Hi David, thank you for driving the release.
> >>> > > > >>
> >>> > > > >> +1 for the 1.15.1 release. The release-1.15 branch
> >>> > > > >> already contains many bug fixes and some SQL
> >>> > > > >> issues are quite critical.
> >>> > > > >>
> >>> > > > >> Btw, FLINK-27606 has been merged just now.
> >>> > > > >>
> >>> > > > >> Best,
> >>> > > > >> Jark
> >>> > > > >>
> >>> > > > >>
> >>> > > > >> On Wed, 8 Jun 2022 at 17:40, David Anderson <
> >>> dander...@apache.org>
> >>> > > > wrote:
> >>> > > > >>
> >>> > > > >>> I would like to start a discussion on releasing 1.15.1. Flink
> >>> 1.15
> >>> > > was
> >>> > > > >>> released on the 5th of May [1] and so far 43 issues have been
> >>> > > resolved,
> >>> > > > >>> including several user-facing issues with blocker and
> critical
> >>> > > > priorities
> >>> > > > >>> [2]. (The recent problem with FileSink rolling policies not
> >>> working
> >>> > > > >>> properly in 1.15.0 got me thinking it might be time for
> bug-fix
> >>> > > > release.)
> >>> > > > >>>
> >>> > > > >>> There currently remain 16 unresolved tickets with a
> fixVersion
> >>> of
> >>> > > > 

Re: [VOTE] Release 1.14.5, release candidate #1

[Konstantin Knauf]

+1 (binding)

* checked signatures and checksums of binaries -> OK
* skimmed over changes since 1.14.4 -> no dependency updates without NOTICE
updates
* reviewed release blog post and -> requested changes

Thanks for preparing this rc!

Konstantin


Am Fr., 10. Juni 2022 um 11:58 Uhr schrieb Xingbo Huang :

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 1.14.5,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 3C2C9FFB59DF9F3E [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.14.5-rc1" [5],
> * website pull request listing the new release and adding announcement blog
> post [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Best,
> Xingbo
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351388
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.14.5-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1509/
> [5] https://github.com/apache/flink/tree/release-1.14.5-rc1
> [6] https://github.com/apache/flink-web/pull/550
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE] Update Flink's Scala 2.12 support from 2.12.7 to 2.12.16

[Konstantin Knauf]

+1 (binding)

Am Fr., 10. Juni 2022 um 18:20 Uhr schrieb Ran Tao :

> +1 non-binding
>
> for that we can't always block at 2.12.7. but we should let users know
> about this binary incompatibility if execute this update.
> btw, the configuration
> of scala-maven-plugin may need to update to match target-8, target-11 jvm
> format.
>
>
> Martijn Visser 于2022年6月10日 周五18:32写道：
>
> > Hi everyone,
> >
> > As previously discussed [1] I would like to open a vote thread for
> updating
> > Flink from Scala 2.12.7 to Scala 2.12.16 and accept that there will be a
> > binary incompatibility when this upgrade is executed.
> >
> > For this vote, I'm following the FLIP criteria since I think that best
> fits
> > with this proposal. That means that the vote will be open for at least 72
> > hours unless there is an objection or not enough votes. Any committer can
> > provide a binding vote.
> >
> > Best regards,
> >
> > Martijn Visser
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
> > [1] https://lists.apache.org/thread/9ft0tbhj45fplt5j3gtb7jzbt4tdr6rh
> >
> --
> Best,
> Ran Tao
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Deprecate SourceFunction APIs

[Konstantin Knauf]

t; > >>> is
> > >>>>> that
> > >>>>>>> part of the DataGeneratorSource?
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Austin
> > >>>>>>>
> > >>>>>>> On Tue, Jun 7, 2022 at 9:34 AM Alexander Fedulov <
> > >>>>>> alexan...@ververica.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi everyone,
> > >>>>>>>>
> > >>>>>>>> Thanks for all the input and a lively discussion. It seems that
> > >>>> there
> > >>>>>> is
> > >>>>>>> a
> > >>>>>>>> consensus that due to
> > >>>>>>>> the inherent complexity of FLIP-27 sources we should provide
> > >> more
> > >>>>>>>> user-facing utilities to bridge
> > >>>>>>>> the gap between the existing SourceFunction-based functionality
> > >>> and
> > >>>>> the
> > >>>>>>> new
> > >>>>>>>> APIs.
> > >>>>>>>>
> > >>>>>>>> To start addressing this I picked the issue that David raised
> > >> and
> > >>>>> many
> > >>>>>>>> upvoted. Here is a proposal
> > >>>>>>>> for  the new DataGeneratorSource: FLIP-238 [1]. Please take a
> > >>>> look, I
> > >>>>>> am
> > >>>>>>>> going to open a separate
> > >>>>>>>> discussion thread on it shortly.
> > >>>>>>>>
> > >>>>>>>> Jing also raised some great points regarding the interfaces and
> > >>>>>>> subclasses.
> > >>>>>>>> It seems to me that
> > >>>>>>>> what might actually help is some sort of a "soft deprecation"
> > >>>> concept
> > >>>>>> and
> > >>>>>>>> annotation. It could be
> > >>>>>>>> used in places where we do not have an alternative
> > >> implementation
> > >>>>> yet,
> > >>>>>>> but
> > >>>>>>>> we clearly want
> > >>>>>>>> to indicate that continuing to build on top of these interfaces
> > >>> is
> > >>>>>>>> discouraged. The area of
> > >>>>>>>> impact of deprecating all SourceFunction subclasses is rather
> > >>> big,
> > >>>>> and
> > >>>>>> we
> > >>>>>>>> can expect it to
> > >>>>>>>> take a while. The hope would be that if in the meantime someone
> > >>>> finds
> > >>>>>>>> themselves using one of
> > >>>>>>>> such old APIs, the "soft deprecation" annotation will be a
> > >> clear
> > >>>>>>> indication
> > >>>>>>>> and encouragement to
> > >>>>>>>> work on introducing an alternative FLIP-27-based implementation
> > >>>>>> instead.
> > >>>>>>>>
> > >>>>>>>> @Austin, in the FLIP I mentioned above [1], the user is
> > >> expected
> > >>> to
> > >>>>>>>> pass a MapFunction > >>>>>>>> OUT>
> > >>>>>>>> to the generator. I wonder if you could have your external
> > >> client
> > >>>> and
> > >>>>>>>> polling logic wrapped in a custom
> > >>>>>>>> MapFunction implementation class? Would that answer your needs
> > >> or
> > >>>> do
> > >>>>>> you
> > >>>>>>>> have some
> > >>>>>>>> more sophisticated scenario in mind?
> > >>>>>>>>
> > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/9Av1D
> > >>>>>>>> Best,
> > >>>>>>>> Alexander Fedulov
> > >>>>>>&g

Re: [DISCUSS] Bi-Weekly Flink Community Sync Meeting

[Konstantin Knauf]

Hi Robert, Hi everyone,

I am still skeptical about this. Generally, orgs of any kind tend to
alternate between "can we do a meeting for this?" and "why don't we do this
async?". For anything that goes towards a status update or information
sharing, asynchronous, written communication is just more inclusive and
efficient. I don't know of why we have the release sync meeting exactly,
but would leave this to the release managers. Should the release sync
actually be recorded like the Kubernetes community meeting? This seems like
a good idea in general.

For an informal get-together, a meeting can be nice, but I would then
separate this from the release sync, and clearly manage expectations what
this meeting is or isn't about and set it up accordingly. For example, for
getting to know each other and informal chatting a big room with more than
6 people becomes awkward very quickly. A series of break-out sessions would
be much more useful then.

Cheers,

Konstantin



Am Do., 2. Juni 2022 um 13:03 Uhr schrieb Robert Metzger <
metrob...@gmail.com>:

> Thanks for your feedback!
>
> Nobody should feel obliged to attend these meetings, or fear that they are
> missing something by not attending. Everything relevant discussed there has
> to be reflected on the mailing list, either as a meeting summary, or in
> existing discussion threads.
> My main motivation is to provide a room for people to get to know each
> other, float some ideas and have informal conversations about Flink.
> Maybe we should call the meeting "Release Sync & Virtual Get Together" or
> something, to manage expectations?
>
> Looking at other projects, this this not uncommon:
>
> Apache Cassandra used to have such meetings for some time in 2020:
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/Cassandra+Contributor+Meeting
>
> (also the K8s SIG:
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/Cassandra+Kubernetes+SIG
> )
>
> The Kubernetes project seems to have quite many meetings from the various
> SIGs:
> - Overview:
>
> https://github.com/kubernetes/community/blob/master/events/community-meeting.md
> - Calendar:
> https://calendar.google.com/calendar/u/0/embed?src=calen...@kubernetes.io
>
> Best,
> Robert
>
>
> On Tue, May 31, 2022 at 2:04 PM Konstantin Knauf 
> wrote:
>
> > Hi everyone,
> >
> > can you be more specific what you mean by "current topics in the Flink
> > Community"? Shouldn't asynchronous communication be the default, and if
> > that doesn't work, we consider a synchronous channel?
> >
> > Cheers,
> >
> > Konstantin
> >
> > Am Di., 31. Mai 2022 um 13:54 Uhr schrieb Jing Ge :
> >
> > > +1
> > > Sounds good! Thanks Robert!
> > >
> > > On Tue, May 31, 2022 at 1:46 PM Márton Balassi <
> balassi.mar...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Robert,
> > > >
> > > > Thanks for the suggestion +1 from me. You already listed the topic of
> > the
> > > > timezone on the wiki that I wanted to bring up.
> > > >
> > > > On Tue, May 31, 2022 at 9:38 AM Robert Metzger 
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > We currently have a bi-weekly release sync meeting on Google Meet
> > every
> > > > > Tuesday at 9am CEST / 3pm China Standard Time / 7am UTC.
> > > > > I would like to propose extending the purpose of the meeting to a
> > > general
> > > > > "Flink Community Sync" meeting, to discuss current topics in the
> > Flink
> > > > > community.
> > > > >
> > > > > I propose that we just collect agenda items on the mailing list in
> > the
> > > > two
> > > > > weeks prior to the meeting.
> > > > > I'm happy to take care of prioritizing agenda items and taking
> notes
> > in
> > > > the
> > > > > wiki.
> > > > > I've created already a page for the next sync:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/2022-06-14+Community+Sync
> > > > >
> > > > > Let me know what you think!
> > > > >
> > > > > Best,
> > > > > Robert
> > > > >
> > > >
> > >
> >
> >
> > --
> > https://twitter.com/snntrable
> > https://github.com/knaufk
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Deprecate SourceFunction APIs

[Konstantin Knauf]

Hi everyone,

very interesting thread. The proposal for deprecation seems to have sparked
a very important discussion. Do we what users struggle with specifically?

Speaking for myself, when I upgrade flink-faker to the new Source API an
unbounded version of the NumberSequenceSource would have been all I needed,
but that's just the data generator use case. I think, that one could be
solved quite easily. David, can you elaborate why you need watermark
generation in the source for your data generators?

Cheers,

Konstantin





Am So., 5. Juni 2022 um 17:48 Uhr schrieb Piotr Nowojski <
pnowoj...@apache.org>:

> Also +1 to what David has written. But it doesn't mean we should be waiting
> indefinitely to deprecate SourceFunction.
>
> Best,
> Piotrek
>
> niedz., 5 cze 2022 o 16:46 Jark Wu  napisał(a):
>
> > +1 to David's point.
> >
> > Usually, when we deprecate some interfaces, we should point users to use
> > the recommended alternatives.
> > However, implementing the new Source interface for some simple scenarios
> is
> > too challenging and complex.
> > We also found it isn't easy to push the internal connector to upgrade to
> > the new Source because
> > "FLIP-27 are hard to understand, while SourceFunction is easy".
> >
> > +1 to make implementing a simple Source easier before deprecating
> > SourceFunction.
> >
> > Best,
> > Jark
> >
> >
> > On Sun, 5 Jun 2022 at 07:29, Jingsong Lee 
> wrote:
> >
> > > +1 to David and Ingo.
> > >
> > > Before deprecate and remove SourceFunction, we should have some easier
> > APIs
> > > to wrap new Source, the cost to write a new Source is too high now.
> > >
> > >
> > >
> > > Ingo Bürk 于2022年6月5日 周日05:32写道：
> > >
> > > > I +1 everything David said. The new Source API raised the complexity
> > > > significantly. It's great to have such a rich, powerful API that can
> do
> > > > everything, but in the process we lost the ability to onboard people
> to
> > > > the APIs.
> > > >
> > > >
> > > > Best
> > > > Ingo
> > > >
> > > > On 04.06.22 21:21, David Anderson wrote:
> > > > > I'm in favor of this, but I think we need to make it easier to
> > > implement
> > > > > data generators and test sources. As things stand in 1.15, unless
> you
> > > can
> > > > > be satisfied with using a NumberSequenceSource followed by a map,
> > > things
> > > > > get quite complicated. I looked into reworking the data generators
> > used
> > > > in
> > > > > the training exercises, and got discouraged by the amount of work
> > > > involved.
> > > > > (The sources used in the training want to be unbounded, and need
> > > > > watermarking in the sources, which means that using
> > > NumberSequenceSource
> > > > > isn't an option.)
> > > > >
> > > > > I think the proposed deprecation will be better received if it can
> be
> > > > > accompanied by something that makes implementing a simple Source
> > easier
> > > > > than it is now. People are continuing to implement new
> > SourceFunctions
> > > > > because the interfaces defined by FLIP-27 are hard to understand,
> > while
> > > > > SourceFunction is easy. Alex, I believe you were looking into
> > > > implementing
> > > > > an easier-to-use building block that could be used in situations
> like
> > > > this.
> > > > > Can we get something like that in place first?
> > > > >
> > > > > David
> > > > >
> > > > > On Fri, Jun 3, 2022 at 4:52 PM Jing Ge  wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> Thanks Alex for driving this!
> > > > >>
> > > > >> +1 To give the Flink developers, especially Connector developers
> the
> > > > clear
> > > > >> signal that the new Source API is recommended according to
> FLIP-27,
> > we
> > > > >> should mark them as deprecated.
> > > > >>
> > > > >> There are some open questions to discuss:
> > > > >>
> > > > >> 1. Do we need to mark all subinterfaces/subclasses as deprecated?
> > e.g.
> > > > >> FromElementsFunction, etc. there are many. What are the
> > replacements?
> > > > >> 2. Do we need to mark all subclasses that have replacement as
> > > > deprecated?
> > > > >> e.g. ExternallyInducedSource whose replacement class, if I am not
> > > > mistaken,
> > > > >> ExternallyInducedSourceReader is @Experimental
> > > > >> 3. Do we need to mark all related test utility classes as
> > deprecated?
> > > > >>
> > > > >> I think it might make sense to create an umbrella ticket to cover
> > all
> > > of
> > > > >> these with the following process:
> > > > >>
> > > > >> 1. Mark SourceFunction as deprecated asap.
> > > > >> 2. Mark subinterfaces and subclasses as deprecated, if there are
> > > > graduated
> > > > >> replacements. Good example is that KafkaSource replaced
> > KafkaConsumer
> > > > which
> > > > >> has been marked as deprecated.
> > > > >> 3. Do not mark subinterfaces and subclasses as deprecated, if
> > > > replacement
> > > > >> classes are still experimental, check if it is time to graduate
> > them.
> > > > After
> > > > >> graduation, go to step 2. It might take a while for graduation.
> > > > >> 

Re: [DISCUSS] Bi-Weekly Flink Community Sync Meeting

[Konstantin Knauf]

Hi everyone,

can you be more specific what you mean by "current topics in the Flink
Community"? Shouldn't asynchronous communication be the default, and if
that doesn't work, we consider a synchronous channel?

Cheers,

Konstantin

Am Di., 31. Mai 2022 um 13:54 Uhr schrieb Jing Ge :

> +1
> Sounds good! Thanks Robert!
>
> On Tue, May 31, 2022 at 1:46 PM Márton Balassi 
> wrote:
>
> > Hi Robert,
> >
> > Thanks for the suggestion +1 from me. You already listed the topic of the
> > timezone on the wiki that I wanted to bring up.
> >
> > On Tue, May 31, 2022 at 9:38 AM Robert Metzger 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > We currently have a bi-weekly release sync meeting on Google Meet every
> > > Tuesday at 9am CEST / 3pm China Standard Time / 7am UTC.
> > > I would like to propose extending the purpose of the meeting to a
> general
> > > "Flink Community Sync" meeting, to discuss current topics in the Flink
> > > community.
> > >
> > > I propose that we just collect agenda items on the mailing list in the
> > two
> > > weeks prior to the meeting.
> > > I'm happy to take care of prioritizing agenda items and taking notes in
> > the
> > > wiki.
> > > I've created already a page for the next sync:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/2022-06-14+Community+Sync
> > >
> > > Let me know what you think!
> > >
> > > Best,
> > > Robert
> > >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE] Creating an Apache Flink slack workspace

[Konstantin Knauf]

+1 for the user community, not the contributor/dev community

Am Do., 19. Mai 2022 um 12:44 Uhr schrieb Yuan Mei :

> +1 (binding)
>
> This facilitates people collaborating on the same project from different
> organizations. I really like this idea.
>
> On Thu, May 19, 2022 at 12:43 PM Peter Huang 
> wrote:
>
> > +1 (non-binding)
> >
> >
> > Best Regards
> > Peter Huang
> >
> > On Wed, May 18, 2022 at 9:33 PM Leonard Xu  wrote:
> >
> > > Thanks Xintong for driving this.
> > >
> > >  +1
> > >
> > > Best,
> > > Leonard
> > >
> > > > 2022年5月19日 上午11:11，Zhou, Brian  写道：
> > > >
> > > > +1 (non-binding)  Slack is a better place for code sharing and quick
> > > discussion.
> > > >
> > > > Regards,
> > > > Brian Zhou
> > > >
> > > > -Original Message-
> > > > From: Yun Tang 
> > > > Sent: Thursday, May 19, 2022 10:32 AM
> > > > To: dev
> > > > Subject: Re: [VOTE] Creating an Apache Flink slack workspace
> > > >
> > > >
> > > > [EXTERNAL EMAIL]
> > > >
> > > > Thanks Xintong for driving this. +1 from my side.
> > > >
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Zhu Zhu 
> > > > Sent: Wednesday, May 18, 2022 17:08
> > > > To: dev 
> > > > Subject: Re: [VOTE] Creating an Apache Flink slack workspace
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Zhu
> > > >
> > > > Timo Walther  于2022年5月18日周三 16:52写道：
> > > >>
> > > >> +1 (binding)
> > > >>
> > > >> Thanks,
> > > >> Timo
> > > >>
> > > >>
> > > >> On 17.05.22 20:44, Gyula Fóra wrote:
> > > >>> +1 (binding)
> > > >>>
> > > >>> On Tue, 17 May 2022 at 19:52, Yufei Zhang 
> > wrote:
> > > >>>
> > >  +1 (nonbinding)
> > > 
> > >  On Tue, May 17, 2022 at 5:29 PM Márton Balassi <
> > > balassi.mar...@gmail.com>
> > >  wrote:
> > > 
> > > > +1 (binding)
> > > >
> > > > On Tue, May 17, 2022 at 11:00 AM Jingsong Li <
> > jingsongl...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Thank Xintong for driving this work.
> > > >>
> > > >> +1
> > > >>
> > > >> Best,
> > > >> Jingsong
> > > >>
> > > >> On Tue, May 17, 2022 at 4:49 PM Martijn Visser <
> > >  martijnvis...@apache.org
> > > >>
> > > >> wrote:
> > > >>
> > > >>> +1 (binding)
> > > >>>
> > > >>> On Tue, 17 May 2022 at 10:38, Yu Li  wrote:
> > > >>>
> > >  +1 (binding)
> > > 
> > >  Thanks Xintong for driving this!
> > > 
> > >  Best Regards,
> > >  Yu
> > > 
> > > 
> > >  On Tue, 17 May 2022 at 16:32, Robert Metzger <
> > metrob...@gmail.com
> > > >
> > > >>> wrote:
> > > 
> > > > Thanks for starting the VOTE!
> > > >
> > > > +1 (binding)
> > > >
> > > >
> > > >
> > > > On Tue, May 17, 2022 at 10:29 AM Jark Wu 
> > >  wrote:
> > > >
> > > >> Thank Xintong for driving this work.
> > > >>
> > > >> +1 from my side (binding)
> > > >>
> > > >> Best,
> > > >> Jark
> > > >>
> > > >> On Tue, 17 May 2022 at 16:24, Xintong Song <
> > > > tonysong...@gmail.com>
> > > > wrote:
> > > >>
> > > >>> Hi everyone,
> > > >>>
> > > >>> As previously discussed in [1], I would like to open a vote
> > >  on
> > >  creating
> > > >> an
> > > >>> Apache Flink slack workspace channel.
> > > >>>
> > > >>> The proposed actions include:
> > > >>> - Creating a dedicated slack workspace with the name Apache
> > > > Flink
> > >  that
> > > > is
> > > >>> controlled and maintained by the Apache Flink PMC
> > > >>> - Updating the Flink website about rules for using various
> > > > communication
> > > >>> channels
> > > >>> - Setting up an Archive for the Apache Flink slack
> > > >>> - Revisiting this initiative by the end of 2022
> > > >>>
> > > >>> The vote will last for at least 72 hours, and will be
> > >  accepted
> > > >> by a
> > > >>> consensus of active PMC members.
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> Xintong
> > > >>>
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > >
> > >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

[Konstantin Knauf]

Hi Lijie,

hm, maybe the following is more appropriate in that case

POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}:merge

Best,

Konstantin

Am Mo., 16. Mai 2022 um 07:05 Uhr schrieb Lijie Wang <
wangdachui9...@gmail.com>:

> Hi Konstantin,
> thanks for your feedback.
>
> From what I understand, PUT should be idempotent. However, we have a
> *timeout* field in the request. This means that initiating the same request
> at two different times will lead to different resource status (timestamps
> of the items to be removed will be different).
>
> Should we use PUT in this case? WDYT?
>
> Best,
> Lijie
>
> Konstantin Knauf  于2022年5月13日周五 17:20写道：
>
> > Hi Lijie,
> >
> > wouldn't the REST API-idiomatic way for an update/replace be a PUT on the
> > resource?
> >
> > PUT: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}
> >
> > Best,
> >
> > Konstantin
> >
> >
> >
> > Am Fr., 13. Mai 2022 um 11:01 Uhr schrieb Lijie Wang <
> > wangdachui9...@gmail.com>:
> >
> > > Hi everyone,
> > >
> > > I've had an offline discussion with Becket Qin and Zhu Zhu, and made
> the
> > > following changes on REST API:
> > > 1. To avoid ambiguity, *timeout* and *endTimestamp* can only choose
> one.
> > If
> > > both are specified, will return error.
> > > 2.  If the specified item is already there, the *ADD* operation has two
> > > behaviors:  *return error*(default value) or *merge/update*, and we
> add a
> > > flag to the request body to control it. You can find more details
> "Public
> > > Interface" section.
> > >
> > > If there is no more feedback, we will start the vote thread next week.
> > >
> > > Best,
> > > Lijie
> > >
> > > Lijie Wang  于2022年5月10日周二 17:14写道：
> > >
> > > > Hi Becket Qin,
> > > >
> > > > Thanks for your suggestions.  I have moved the description of
> > > > configurations, metrics and REST API into "Public Interface" section,
> > and
> > > > made a few updates according to your suggestion.  And in this FLIP,
> > there
> > > > no public java Interfaces or pluggables that users need to implement
> by
> > > > themselves.
> > > >
> > > > Answers for you questions:
> > > > 1. Yes, there 2 block actions: MARK_BLOCKED and.
> > > > MARK_BLOCKED_AND_EVACUATE_TASKS (has renamed). Currently, block items
> > can
> > > > only be added through the REST API, so these 2 action are mentioned
> in
> > > the
> > > > REST API part (The REST API part has beed moved to public interface
> > now).
> > > > 2. I agree with you. I have changed the "Cause" field to String, and
> > > allow
> > > > users to specify it via REST API.
> > > > 3. Yes, it is useful to allow different timeouts. As mentioned above,
> > we
> > > > will introduce 2 fields : *timeout* and *endTimestamp* into the ADD
> > REST
> > > > API to specify when to remove the blocked item. These 2 fields are
> > > > optional, if neither is specified, it means that the blocked item is
> > > > permanent and will not be removed. If both are specified, the minimum
> > of
> > > > *currentTimestamp+tiemout *and* endTimestamp* will be used as the
> time
> > to
> > > > remove the blocked item. To keep the configurations more minimal, we
> > have
> > > > removed the *cluster.resource-blocklist.item.timeout* configuration
> > > > option.
> > > > 4. Yes, the block item will be overridden if the specified item
> already
> > > > exists. The ADD operation is *ADD or UPDATE*.
> > > > 5. Yes. On JM/RM side, all the blocklist information is maintained in
> > > > JMBlocklistHandler/RMBlocklistHandler. The blocklist handler(or
> > > abstracted
> > > > to other interfaces) will be propagated to different components.
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Becket Qin  于2022年5月10日周二 11:26写道：
> > > >
> > > >> Hi Lijie,
> > > >>
> > > >> Thanks for updating the FLIP. It looks like the public interface
> > section
> > > >> did not fully reflect all the user sensible behavior and API. Can
> you
> > > put
> > > >> everything that users may be aware of there? That would include the
> > REST
> > >

Re: [ANNOUNCE] Problems with build due to tests using too much disk space

[Konstantin Knauf]

Thanks a lot for taking care of this, Martijn!

Am Fr., 13. Mai 2022 um 12:59 Uhr schrieb Martijn Visser <
martijnvis...@apache.org>:

> Hi everyone,
>
> I noticed that multiple e2e tests had debugging logged enabled. Quite a few
> of them have done that for a long time; however, since we are adding more
> and more tests, it also seems that this caused the bucket to overflood.
> I've disabled the debugging logging for all tests; this has resolved the
> issue (for now).
>
> Best regards,
>
> Martijn
>
> On Thu, 12 May 2022 at 15:39, Martijn Visser 
> wrote:
>
> > Hi everyone,
> >
> > There are currently problems with builds on master, release-1.15 and
> > release-1.14. This is tracked under
> > https://issues.apache.org/jira/browse/FLINK-24433
> >
> > The E2E tests are failing due to running out of disk space. I have
> noticed
> > something in one of the bash based E2E tests (where a TAR is created, but
> > never removed). If there's anyone else who wants to help/contribute in
> > fixing this, this is much appreciated.
> >
> > I'll keep you posted.
> >
> > Best regards,
> >
> > Martijn Visser
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

[Konstantin Knauf]

 and then
> >> > > > > >> > >> > take the command from that service to block the
> >> problematic
> >> > > > > nodes.
> >> > > > > >> On
> >> > > > > >> > the
> >> > > > > >> > >> > other hand, for users who do not have such a service,
> >> > simply
> >> > > > > >> letting
> >> > > > > >> > Flink
> >> > > > > >> > >> > be clever by itself to block the suspicious nodes
> might
> >> be
> >> > > > > desired
> >> > > > > >> to
> >> > > > > >> > >> > ensure the jobs are running smoothly.
> >> > > > > >> > >> >
> >> > > > > >> > >> > So that indicates a) and b) here should be pluggable /
> >> > > > optional.
> >> > > > > >> > >> >
> >> > > > > >> > >> > In light of this, maybe it would make sense to have
> >> > something
> >> > > > > >> > pluggable
> >> > > > > >> > >> > like a UnstableNodeReporter which exposes unstable
> nodes
> >> > > > > actively.
> >> > > > > >> (A
> >> > > > > >> > more
> >> > > > > >> > >> > general interface should be JobInfoReporter which
> >> can be
> >> > > > used
> >> > > > > to
> >> > > > > >> > report
> >> > > > > >> > >> > any information of type . But I'll just keep the
> >> scope
> >> > > > > relevant
> >> > > > > >> to
> >> > > > > >> > this
> >> > > > > >> > >> > FLIP here). Personally speaking, I think it is OK to
> >> have a
> >> > > > > default
> >> > > > > >> > >> > implementation of a reporter which just tells Flink to
> >> take
> >> > > > > action
> >> > > > > >> to
> >> > > > > >> > block
> >> > > > > >> > >> > problematic nodes and also unblocks them after
> timeout.
> >> > > > > >> > >> >
> >> > > > > >> > >> > Thanks,
> >> > > > > >> > >> >
> >> > > > > >> > >> > Jiangjie (Becket) Qin
> >> > > > > >> > >> >
> >> > > > > >> > >> >
> >> > > > > >> > >> > On Mon, May 2, 2022 at 3:27 PM Роман Бойко <
> >> > > > ro.v.bo...@gmail.com
> >> > > > > >
> >> > > > > >> > wrote:
> >> > > > > >> > >> >
> >> > > > > >> > >> > > Thanks for good initiative, Lijie and Zhu!
> >> > > > > >> > >> > >
> >> > > > > >> > >> > > If it's possible I'd like to participate in
> >> development.
> >> > > > > >> > >> > >
> >> > > > > >> > >> > > I agree with 3rd point of Konstantin's reply - we
> >> should
> >> > > > > consider
> >> > > > > >> > to move
> >> > > > > >> > >> > > somehow the information of blocklisted nodes/TMs
> from
> >> > > active
> >> > > > > >> > >> > > ResourceManager to non-active ones. Probably storing
> >> > inside
> >> > > > > >> > >> > > Zookeeper/Configmap might be helpful here.
> >> > > > > >> > >> > >
> >> > > > > >> > >> > > And I agree with Martijn that a lot of organizations
> >> > don't
> >> > > > want
> >> > > > > >> to
> >> > > > > >> > expose
> >> > > > > >> > >> > > such API for a cluster user group. But I think it's
> >> > > necessary

Re: [DISCUSS] Release first version of Elasticsearch connector from external repository

[Konstantin Knauf]

Hi Martijn,

+1 to do a release which is compatible with Flink 1.5.x. With the release,
we should add something like a compatibility matrix to the documentation,
but I am sure that's already on your list.

Cheers,

Konstantin




Am Mi., 11. Mai 2022 um 20:10 Uhr schrieb Martijn Visser <
martijnvis...@apache.org>:

> Hi everyone,
>
> As mentioned in the previous update [1] we were working on the final step
> for moving out the Elasticsearch connector, which was related to the
> documentation integration. This has now been completed: the documentation
> you see on
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/
> is generated via the documentation in the Elasticsearch external
> repository.
>
> In our plan we outlined that when we moved things out, we would create a
> first release out of this repository [2]. I would like to take that step to
> complete this first connector and move on to the next one.
>
> If there's any feedback or questions, do let me know. Else I'll reach out
> to some PMCs to help facilitate this release and we'll open a VOTE thread
> shortly.
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
> [1] https://lists.apache.org/thread/8k1xonqt7hn0xldbky1cxfx3fzh6sj7h
> [2] https://lists.apache.org/thread/vqbjoo94wwqcvo32c80dkmp7r5gmy68r
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [Discuss] Creating an Apache Flink slack workspace

[Konstantin Knauf]

I don't think we can maintain two additional channels. Some people have
already concerns about covering one additional channel.

I think, a forum provides a better user experience than a mailing list.
Information is structured better, you can edit messages, sign up and search
is easier.

To make some progress, maybe we decide on chat vs forum vs none and then go
into a deeper discussion on the implementation or is there anything about
Slack that would be complete blocker for the implementation?



Am Mi., 11. Mai 2022 um 07:35 Uhr schrieb Xintong Song <
tonysong...@gmail.com>:

> I agree with Robert on reworking the "Community" and "Getting Help" pages
> to emphasize how we position the mailing lists and Slack, and on revisiting
> in 6-12 months.
>
> Concerning dedicated Apache Flink Slack vs. ASF Slack, I'm with
> Konstantin. I'd expect it to be easier for having more channels and keeping
> them organized, managing permissions for different roles, adding bots, etc.
>
> IMO, having Slack is about improving the communication efficiency when you
> are already in a discussion, and we expect such improvement would motivate
> users to interact more with each other. From that perspective, forums are
> not much better than mailing lists.
>
> I'm also open to forums as well, but not as an alternative to Slack. I
> definitely see how forums help in keeping information organized and easy to
> find. However, I'm a bit concerned about the maintenance overhead. I'm not
> very familiar with Discourse or Reddit. My impression is that they are not
> as easy to set up and maintain as Slack.
>
> Thank you~
>
> Xintong Song
>
>
> [1] https://asktug.com/
>
> On Tue, May 10, 2022 at 4:50 PM Konstantin Knauf 
> wrote:
>
>> Thanks for starting this discussion again. I am pretty much with Timo
>> here. Slack or Discourse as an alternative for the user community, and
>> mailing list for the contributing, design discussion, etc. I definitely see
>> the friction of joining a mailing list and understand if users are
>> intimidated.
>>
>> I am leaning towards a forum aka Discourse over a chat aka Slack. This is
>> about asking for help, finding information and thoughtful discussion more
>> so than casual chatting, right? For this a forum, where it is easier to
>> find and comment on older threads and topics just makes more sense to me. A
>> well-done Discourse forum is much more inviting and vibrant than a mailing
>> list. Just from a tool perspective, discourse would have the advantage of
>> being Open Source and so we could probably self-host it on an ASF machine.
>> [1]
>>
>> When it comes to Slack, I definitely see the benefit of a dedicated
>> Apache Flink Slack compared to ASF Slack. For example, we could have more
>> channels (e.g. look how many channels Airflow is using
>> http://apache-airflow.slack-archives.org) and we could generally
>> customize the experience more towards Apache Flink.  If we go for Slack,
>> let's definitely try to archive it like Airflow did. If we do this, we
>> don't necessarily need infinite message retention in Slack itself.
>>
>> Cheers,
>>
>> Konstantin
>>
>> [1] https://github.com/discourse/discourse
>>
>>
>> Am Di., 10. Mai 2022 um 10:20 Uhr schrieb Timo Walther <
>> twal...@apache.org>:
>>
>>> I also think that a real-time channel is long overdue. The Flink
>>> community in China has shown that such a platform can be useful for
>>> improving the collaboration within the community. The DingTalk channel of
>>> 10k+ users collectively helping each other is great to see. It could also
>>> reduce the burden from committers for answering frequently asked questions.
>>>
>>> Personally, I'm a mailing list fan esp. when it comes to design
>>> discussions. In my opinion, the dev@ mailing list should definitely
>>> stay where and how it is. However, I understand that users might not want
>>> to subscribe to a mailing list for a single question and get their mailbox
>>> filled with unrelated discussions afterwards. Esp. in a company setting it
>>> might not be easy to setup a dedicated email address for mailing lists and
>>> setting up rules is also not convenient.
>>>
>>> It would be great if we could use the ASF Slack. We should find an
>>> official, accessible channel. I would be open for the right tool. It might
>>> make sense to also look into Discourse or even Reddit? The latter would
>>> definitely be easier to index by a search engine. Discourse is actually
>>> made for modern real-time forums.
>>>
>>&g

Re: flink Job is throwing depdnecy issue when submitted to clusrer

[Konstantin Knauf]

Hi there,

are you using any of Flink S3 Filesystems? If so, where do you load it from

a) lib/
b) plugins/
c) bundled with your Job in a fat JAR

b) would be the right way to do it in Flink 1.13. I don't know if this
fixes the issue, but IIRC because we introduced the plugin mechansim we
don't relocated dependencies in the filesystems anymore.

Cheers,

Konstantin





Am Sa., 7. Mai 2022 um 07:47 Uhr schrieb 张立志 :

> 退订
>
>
>
> | |
> zh_ha...@163.com
> |
> |
> 邮箱：zh_ha...@163.com
> |
>
>
>
>
>  回复的原邮件 
> | 发件人 | Great Info |
> | 日期 | 2022年05月07日 13:21 |
> | 收件人 | dev@flink.apache.org、user<
> u...@flink.apache.org> |
> | 抄送至 | |
> | 主题 | flink Job is throwing depdnecy issue when submitted to clusrer |
> I have one flink job which reads files from s3 and processes them.
> Currently, it is running on flink 1.9.0, I need to upgrade my cluster to
> 1.13.5, so I have done the changes in my job pom and brought up the flink
> cluster using 1.13.5 dist.
>
> when I submit my application I am getting the below error when it tries to
> connect to s3, have updated the s3 SDK version to the latest, but still
> getting the same error.
>
> caused by: java.lang.invoke.lambdaconversionexception: invalid receiver
> type interface org.apache.http.header; not a subtype of implementation type
> interface org.apache.http.namevaluepair
>
> it works when I just run as a mini-cluster ( running just java -jar
> ) and also when I submit to the Flink cluster with 1.9.0.
>
> Not able to understand where the dependency match is happening.
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: JobManager High Availability

[Konstantin Knauf]

Hi Matyas,

yes, that's expected. The feature should have never been called "high
availability", but something like "Flink Jobmanager failover", because
that's what it is.

With standby Jobmanagers what you gain is a faster failover, because a new
Jobmanager does not need to be started before restarting the Job. That's
all.

Cheers,

Konstantin

Am Di., 10. Mai 2022 um 10:56 Uhr schrieb Őrhidi Mátyás <
matyas.orh...@gmail.com>:

> Hi Folks!
>
> I've been goofing around with the JobManager HA configs using multiple JM
> replicas (in the Flink Kubernetes Operator). It's working seemingly fine,
> however the job itself is being restarted when you kill the leader JM pod.
> Is this expected?
>
> Thanks,
> Matyas
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [Discuss] Creating an Apache Flink slack workspace

[Konstantin Knauf]

Thanks for starting this discussion again. I am pretty much with Timo here.
Slack or Discourse as an alternative for the user community, and mailing
list for the contributing, design discussion, etc. I definitely see the
friction of joining a mailing list and understand if users are intimidated.

I am leaning towards a forum aka Discourse over a chat aka Slack. This is
about asking for help, finding information and thoughtful discussion more
so than casual chatting, right? For this a forum, where it is easier to
find and comment on older threads and topics just makes more sense to me. A
well-done Discourse forum is much more inviting and vibrant than a mailing
list. Just from a tool perspective, discourse would have the advantage of
being Open Source and so we could probably self-host it on an ASF machine.
[1]

When it comes to Slack, I definitely see the benefit of a dedicated Apache
Flink Slack compared to ASF Slack. For example, we could have more channels
(e.g. look how many channels Airflow is using
http://apache-airflow.slack-archives.org) and we could generally customize
the experience more towards Apache Flink.  If we go for Slack, let's
definitely try to archive it like Airflow did. If we do this, we don't
necessarily need infinite message retention in Slack itself.

Cheers,

Konstantin

[1] https://github.com/discourse/discourse


Am Di., 10. Mai 2022 um 10:20 Uhr schrieb Timo Walther :

> I also think that a real-time channel is long overdue. The Flink community
> in China has shown that such a platform can be useful for improving the
> collaboration within the community. The DingTalk channel of 10k+ users
> collectively helping each other is great to see. It could also reduce the
> burden from committers for answering frequently asked questions.
>
> Personally, I'm a mailing list fan esp. when it comes to design
> discussions. In my opinion, the dev@ mailing list should definitely stay
> where and how it is. However, I understand that users might not want to
> subscribe to a mailing list for a single question and get their mailbox
> filled with unrelated discussions afterwards. Esp. in a company setting it
> might not be easy to setup a dedicated email address for mailing lists and
> setting up rules is also not convenient.
>
> It would be great if we could use the ASF Slack. We should find an
> official, accessible channel. I would be open for the right tool. It might
> make sense to also look into Discourse or even Reddit? The latter would
> definitely be easier to index by a search engine. Discourse is actually
> made for modern real-time forums.
>
> Regards,
> Timo
>
>
> Am 10.05.22 um 09:59 schrieb David Anderson:
>
> Thank you @Xintong Song  for sharing the
> experience of the Flink China community.
>
> I'm become convinced we should give Slack a try, both for discussions
> among the core developers, and as a place where the community can reach out
> for help. I am in favor of using the ASF slack, as we will need a paid
> instance for this to go well, and joining it is easy enough (took me about
> 2 minutes). Thanks, Robert, for suggesting we go down this route.
>
> David
>
> On Tue, May 10, 2022 at 8:21 AM Robert Metzger 
> wrote:
>
>> It seems that we'd have to use invite links on the Flink website for
>> people to join our Slack (1)
>> These links can be configured to have no time-expiration, but they will
>> expire after 100 guests have joined.
>> I guess we'd have to use a URL shortener (https://s.apache.org) that we
>> update once the invite link expires. It's not a nice solution, but it'll
>> work.
>>
>>
>> (1) https://the-asf.slack.com/archives/CBX4TSBQ8/p1652125017094159
>>
>>
>> On Mon, May 9, 2022 at 3:59 PM Robert Metzger 
>> wrote:
>>
>>> Thanks a lot for your answer. The onboarding experience to the ASF Slack
>>> is indeed not ideal:
>>> https://apisix.apache.org/docs/general/join#join-the-slack-channel
>>> I'll see if we can improve it
>>>
>>> On Mon, May 9, 2022 at 3:38 PM Martijn Visser 
>>> wrote:
>>>
 As far as I recall you can't sign up for the ASF instance of Slack, you
 can
 only get there if you're a committer or if you're invited by a
 committer.

 On Mon, 9 May 2022 at 15:15, Robert Metzger 
 wrote:

 > Sorry for joining this discussion late, and thanks for the summary
 Xintong!
 >
 > Why are we considering a separate slack instance instead of using the
 ASF
 > Slack instance?
 > The ASF instance is paid, so all messages are retained forever, and
 quite
 > a few people are already on that Slack instance.
 > There is already a #flink channel on that Slack instance, that we
 could
 > leave as passive as it is right now, or put some more effort into it,
 on a
 > voluntary basis.
 > We could add another #flink-dev channel to that Slack for developer
 > discussions, and a private flink-committer and flink-pmc chat.
 >
 > If we are going that path, we should rework 

Re: [DISCUSS] DockerHub repository maintainers

[Konstantin Knauf]

Hi Xintong,

it is a pity that we can only have 5 maintainers. Every (patch) release of
flink, flink-statefun, the flink-kubernetes-operator requires a maintainer
to publish the image then, if I am not mistaken. As its mostly different
groups managing the sub-projects, this is quite the bottleneck. If we give
one seat to flink-statefun maintainers, one to the
flink-kubernetes-operator maintainers, this leaves three seats for Apache
Flink core, and there is no redundancy for the other projects. When I
managed the last two patch releases, the DockerHub access was also the
biggest hurdle. Maybe we can talk to the INFRA people again. We can
certainly reduce it, but 5 is very little.

Cheers,

Konstantin






Am Fr., 6. Mai 2022 um 09:00 Uhr schrieb Till Rohrmann :

> Hi everyone,
>
> thanks for starting this discussion Xintong. I would volunteer as a
> maintainer of the flink-statefun Docker repository if you need one.
>
> Cheers,
> Till
>
> On Fri, May 6, 2022 at 6:22 AM Xintong Song  wrote:
>
>> It seems to me we at least don't have a consensus on dropping the use of
>> apache namespace, which means we need to decide on a list of maintainers
>> anyway. So maybe we can get the discussion back to the maintainers. We may
>> continue the official-image vs. apache-namespace in a separate thread if
>> necessary.
>>
>> As mentioned previously, we need to reduce the number of maintainers from
>> 20 to 5, as required by INFRA. Jingsong and I would like to volunteer as 2
>> of the 5, and we would like to learn who else wants to join us. Of course
>> the list of maintainers can be modified later.
>>
>> *This also means the current maintainers may be removed from the list.*
>> Please let us know if you still need that privilege. CC-ed all the current
>> maintainers for attention.
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>> On Wed, May 4, 2022 at 3:14 PM Chesnay Schepler 
>> wrote:
>>
>> > One advantage is that the images are periodically rebuilt to get
>> > security fixes.
>> >
>> > The operator is a different story anyway because it is AFAIK only
>> > supposed to be used via docker
>> > (i.e., no standalone mode), which alleviates concerns about keeping the
>> > logic within the image
>> > to a minimum (which bit us in the past on the flink side).
>> >
>> > On 03/05/2022 16:09, Yang Wang wrote:
>> > > The flink-kubernetes-operator project is only published
>> > > via apache/flink-kubernetes-operator on docker hub and github
>> packages.
>> > > We do not find the obvious advantages by using docker hub official
>> > images.
>> > >
>> > > Best,
>> > > Yang
>> > >
>> > > Xintong Song  于2022年4月28日周四 19:27写道：
>> > >
>> > >> I agree with you that doing QA for the image after the release has
>> been
>> > >> finalized doesn't feel right. IIUR, that is mostly because official
>> > image
>> > >> PR needs 1) the binary release being deployed and propagated and 2)
>> the
>> > >> corresponding git commit being specified. I'm not completely sure
>> about
>> > >> this. Maybe we can improve the process by investigating more about
>> the
>> > >> feasibility of pre-verifying an official image PR before finalizing
>> the
>> > >> release. It's definitely a good thing to do if possible.
>> > >>
>> > >> I also agree that QA from DockerHub folks is valuable to us.
>> > >>
>> > >> I'm not against publishing official-images, and I'm not against
>> working
>> > >> closely with the DockerHub folks to improve the process of delivering
>> > the
>> > >> official image. However, I don't think these should become reasons
>> that
>> > we
>> > >> don't release our own apache/flink images.
>> > >>
>> > >> Taking the 1.12.0 as an example, admittedly it would be nice for us
>> to
>> > >> comply with the DockerHub folks' standards and not have a
>> > >> just-for-kubernetes command in our entrypoint. However, this is IMO
>> far
>> > >> less important compared to delivering the image to our users timely.
>> I
>> > >> guess that's where the DockerHub folks and us have different
>> > >> priorities, and that's why I think we should have a path that is
>> fully
>> > >> controlled by this community to deliver images. We could take their
>> > >> valuable inputs and improve afterwards. Actually, that's what we did
>> for
>> > >> 1.12.0 by starting to release to apache/flink.
>> > >>
>> > >> Thank you~
>> > >>
>> > >> Xintong Song
>> > >>
>> > >>
>> > >>
>> > >> On Thu, Apr 28, 2022 at 6:30 PM Chesnay Schepler > >
>> > >> wrote:
>> > >>
>> > >>> I still think that's mostly a process issue.
>> > >>> Of course we can be blind-sided if we do the QA for a release
>> artifact
>> > >>> after the release has been finalized.
>> > >>> But that's a clearly broken process from the get-go.
>> > >>>
>> > >>> At the very least we should already open a PR when the RC is
>> created to
>> > >>> get earlier feedback.
>> > >>>
>> > >>> Moreover, nowadays the docker images are way slimmer and we are much
>> > >>> more careful on what is actually added to the scripts.
>> > >>>
>> > 

Re: [DISCUSS] Planning Flink 1.16

[Konstantin Knauf]

Hi everyone,

Xingbo, Becket, Chesnay, Martijn, Godfrey and myself briefly talked offline
and we would propose that Xingbo, Godfrey, Martijn and Chesnay take on the
release management for the next release. Six people seemed like an overkill
but having a bit of high availability in both Europe and Asia sounded like
a good dea. These four will follow up with next steps.

Thank you,

Konstantin



Am Fr., 29. Apr. 2022 um 15:17 Uhr schrieb Becket Qin :

> Thanks for kicking off the topic, Konstantin and Chesnay.
>
> Also thanks Martijn, Godfrey and Xingbo for volunteering to be the release
> manager. Given that release 1.16 would likely be a beefy release with a
> bunch of major features already on their way, it might make sense to have
> more release managers than we usually do. We can figure out how to
> collaborate, e.g. splitting by modules / FLIPs. In case we need someone to
> get some errands or coordination done, I am happy to help.
>
> Also, +1 for feature freeze by the end of July with a potential 2-week
> delay of contingency.
>
> Cheers,
>
> Jiangjie (Becket) Qin
>
> On Fri, Apr 29, 2022 at 5:14 PM Xingbo Huang  wrote:
>
> > Thanks Konstantin and Chesnay for starting the discussion. I'm also
> willing
> > to volunteer as the release manager if this is still open.
> >
> > Regarding the feature freeze date, +1 to the end of mid August.
> >
> > Best,
> > Xingbo
> >
> > Zhu Zhu  于2022年4月29日周五 11:01写道：
> >
> > > +1 for a 5 months release cycle.
> > > +1 target the feature freeze date 1.5 months before the release date.
> It
> > > can better guard the release date.
> > > Therefore, also +1 for mid August as the feature freeze date of 1.16
> > >
> > > Thanks,
> > > Zhu
> > >
> > > Jark Wu  于2022年4月28日周四 22:24写道：
> > >
> > > > I'm also fine with the end of July for the feature freeze.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Thu, 28 Apr 2022 at 21:00, Martijn Visser 
> > > > wrote:
> > > >
> > > > > +1 for continuing to strive for a 5 months release cycle.
> > > > >
> > > > > And +1 to have the planned feature freeze mid August, which I would
> > > > propose
> > > > > to have happen on Monday the 15th of August 2022. I would also
> > already
> > > > > state that even though we know this is a holiday period, we should
> > not
> > > > > extend this deadline for that reason :)
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn Visser
> > > > > https://twitter.com/MartijnVisser82
> > > > > https://github.com/MartijnVisser
> > > > >
> > > > >
> > > > > On Thu, 28 Apr 2022 at 14:37, Jingsong Li 
> > > > wrote:
> > > > >
> > > > > > Thanks for the check.
> > > > > >
> > > > > > > Continue to strive for a 5 months release cycle and 1.5 months
> > > before
> > > > > to
> > > > > > the desired release date
> > > > > >
> > > > > > Sounds good to me!
> > > > > >
> > > > > > Best,
> > > > > > Jingsong
> > > > > >
> > > > > > On Thu, Apr 28, 2022 at 7:06 PM Konstantin Knauf <
> > kna...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > thank you for the feedback so far. I've checked the length of
> the
> > > > > > previous
> > > > > > > last release cycles. Up until Flink 1.14, we've actually been
> > > > > incredibly
> > > > > > > good at maintaining a 5 months release cycle (see below).
> > > > > Interestingly,
> > > > > > > the community is officially targeting a 4 months release cycle
> > [1].
> > > > > > >
> > > > > > > - 1.15.0 2022-05-01? (7 months, 2 days?)
> > > > > > > - 1.14.0: 2021-09-29 (4 months, 26 days)
> > > > > > > - 1.13.0: 2021-05-03 (4 months, 23 days)
> > > > > > > - 1.12.0: 2020-12-10 (5 months, 3 days)
> > > > > > > - 1.11.0: 2020-07-07 (4 months, 26 days)
> > > > > > > - 1.10.0: 2020-02-11
> > > > > > &g

Re: [ANNOUNCE] Apache Flink 1.15.0 released

[Konstantin Knauf]

Congratulations to everyone!

Thanks to Yun Gao, Joe Moser & Till Rohrmann for managing this release.
It's done :)

Am Do., 5. Mai 2022 um 09:57 Uhr schrieb Xintong Song :

> Congratulations~!
>
> Thank the release managers, and thank all who contributed to this release.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, May 5, 2022 at 3:45 PM Guowei Ma  wrote:
>
> > Hi, Yun
> >
> > Great job!
> > Thank you very much for your efforts to release Flink-1.15 during this
> > time.
> > Thanks also to all the contributors who worked on this release!
> >
> > Best,
> > Guowei
> >
> >
> > On Thu, May 5, 2022 at 3:24 PM Peter Schrott 
> > wrote:
> >
> > > Great!
> > >
> > > Will install it on the cluster asap! :)
> > >
> > > One thing I noticed: the linked release notes in the blog announcement
> > > under "Upgrade Notes" result in a 404
> > > (
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.15/release-notes/flink-1.15/
> > > )
> > >
> > > They are also not linked on the main page:
> > > https://nightlies.apache.org/flink/flink-docs-release-1.15/
> > >
> > > Keep it up!
> > > Peter
> > >
> > >
> > > On Thu, May 5, 2022 at 8:43 AM Martijn Visser 
> > > wrote:
> > >
> > > > Thank you Yun Gao, Till and Joe for driving this release. Your
> efforts
> > > are
> > > > greatly appreciated!
> > > >
> > > > To everyone who has opened Jira tickets, provided PRs, reviewed code,
> > > > written documentation or anything contributed in any other way, this
> > > > release was (once again) made possible by you! Thank you.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > Op do 5 mei 2022 om 08:38 schreef Yun Gao 
> > > >
> > > >> The Apache Flink community is very happy to announce the release of
> > > >> Apache Flink 1.15.0, which is the first release for the Apache Flink
> > > >> 1.15 series.
> > > >>
> > > >> Apache Flink® is an open-source stream processing framework for
> > > >> distributed, high-performing, always-available, and accurate data
> > > >> streaming applications.
> > > >>
> > > >> The release is available for download at:
> > > >> https://flink.apache.org/downloads.html
> > > >>
> > > >> Please check out the release blog post for an overview of the
> > > >> improvements for this release:
> > > >> https://flink.apache.org/news/2022/05/05/1.15-announcement.html
> > > >>
> > > >> The full release notes are available in Jira:
> > > >>
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350442
> > > >>
> > > >> We would like to thank all contributors of the Apache Flink
> community
> > > >> who made this release possible!
> > > >>
> > > >> Regards,
> > > >> Joe, Till, Yun Gao
> > > >>
> > > > --
> > > >
> > > > Martijn Visser | Product Manager
> > > >
> > > > mart...@ververica.com
> > > >
> > > > 
> > > >
> > > >
> > > > Follow us @VervericaData
> > > >
> > > > --
> > > >
> > > > Join Flink Forward  - The Apache Flink
> > > > Conference
> > > >
> > > > Stream Processing | Event Driven | Real Time
> > > >
> > > >
> > >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE] Apache Flink Table Store 0.1.0, release candidate #2

[Konstantin Knauf]

+1 (binding)

* mvn clean package - PASSED
* checked signatures & checksums of source artifacts - OK
* went through quick start - WORKS
* skimmed over NOTICE file - LOOKS GOOD

I also read over the announcement blog post. In my opinion, we could try to
motivate the project a bit better. What is the problem that
flink-table-store will solve? What is its relationship to Apache Flink
itself? Since this is the first release, I am sure a lot of users will ask
themselves these questions.





Am Fr., 29. Apr. 2022 um 11:49 Uhr schrieb Caizhi Weng :

> Hi all!
>
> +1 for the release (non-binding). I've tested the jar with a standalone
> cluster and SQL client.
>
>- Compiled the sources;
>- Run through quick start guide;
>- Test all supported data types;
>- Check that table store jar has no conflict with orc / avro format and
>kafka connector;
>- Check that both file store and log store works.
>
> I haven't checked the signatures as my gnupg version is old.
>
> Jingsong Li  于2022年4月29日周五 10:24写道：
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #2 for the version 0.1.0
> of
> > Apache Flink Table Store, as follows:
> >
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > **Release Overview**
> >
> > As an overview, the release consists of the following:
> > a) Table Store canonical source distribution, to be deployed to the
> > release repository at dist.apache.org
> > b) Maven artifacts to be deployed to the Maven Central Repository
> >
> > **Staging Areas to Review**
> >
> > The staging areas containing the above mentioned artifacts are as
> follows,
> > for your review:
> > * All artifacts for a) and b) can be found in the corresponding dev
> > repository at dist.apache.org [2]
> > * All artifacts for c) can be found at the Apache Nexus Repository [3]
> > * Pre Bundled Binaries Jar can work fine with quick start [4][5]
> >
> > All artifacts are signed with the key
> > 2C2B6A653B07086B65E4369F7C76245E0A318150 [6]
> >
> > Other links for your review:
> > * JIRA release notes [7]
> > * source code tag "release-0.1.0-rc2" [8]
> > * PR to update the website Downloads page to include Table Store
> > links [9]
> >
> > **Vote Duration**
> >
> > The voting time will run for at least 72 hours.
> > It is adopted by majority approval, with at least 3 PMC affirmative
> votes.
> >
> > Best,
> > Jingsong Lee
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Table+Store+Release
> > [2]
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-table-store-0.1.0-rc2/
> > [3]
> > https://repository.apache.org/content/repositories/orgapacheflink-1502/
> > [4]
> >
> https://repository.apache.org/content/repositories/orgapacheflink-1502/org/apache/flink/flink-table-store-dist/0.1.0/flink-table-store-dist-0.1.0.jar
> > [5]
> >
> https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1/docs/try-table-store/quick-start/
> > [6] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [7]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351234
> > [8] https://github.com/apache/flink-table-store/tree/release-0.1.0-rc2
> > [9] https://github.com/apache/flink-web/pull/531
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Planning Flink 1.16

[Konstantin Knauf]

Hi everyone,

thank you for the feedback so far. I've checked the length of the previous
last release cycles. Up until Flink 1.14, we've actually been incredibly
good at maintaining a 5 months release cycle (see below). Interestingly,
the community is officially targeting a 4 months release cycle [1].

- 1.15.0 2022-05-01? (7 months, 2 days?)
- 1.14.0: 2021-09-29 (4 months, 26 days)
- 1.13.0: 2021-05-03 (4 months, 23 days)
- 1.12.0: 2020-12-10 (5 months, 3 days)
- 1.11.0: 2020-07-07 (4 months, 26 days)
- 1.10.0: 2020-02-11

The 1.15 release cycle has took significantly longer. In my opinion we
should try to get back into the 5 months cadence with the next release.
Since empirically we always often end up moving the the feature freeze by a
week or two, and that we often need about a month for release testing &
stabilization and releasing, I don't think, we should move the planned
feature freeze to later than
*mid August. *
What do you think:
1. Should we continue to strive for a 5 months release cycle (and update
[1] accordingly)?
2. Does it sound reasonable to target a feature freeze date, which is 1.5
months before to the desired release date?

Cheers,

Konstantin

 [1] https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases

Am Do., 28. Apr. 2022 um 05:20 Uhr schrieb Jingsong Li <
jingsongl...@gmail.com>:

> Thanks Konstantin and Chesnay for starting the discussion. And thanks
> Konstantin, Chesnay, Godfrey, Martijn for volunteering.
>
> The 1.16 release at the end of July means that there are currently
> only 3 months to fully commit to 1.16 development, and I'm a little
> concerned that this is too short a time frame, which may result in
> some features only reaching a halfway decent state.
>
> Best,
> Jingsong
>
>
> On Wed, Apr 27, 2022 at 7:10 PM David Morávek  wrote:
> >
> > Thanks Konstantin and Chesnay for starting the discussion and
> volunteering.
> > The timeline proposal sounds reasonable :+1:
> >
> > Best,
> > D.
> >
> > On Tue, Apr 26, 2022 at 1:37 PM Martijn Visser  >
> > wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for starting this discussion. I would also volunteer to help
> out as
> > > a release manager for the 1.16 release.
> > >
> > > Best regards,
> > >
> > > Martijn Visser
> > > https://twitter.com/MartijnVisser82
> > > https://github.com/MartijnVisser
> > >
> > >
> > > On Tue, 26 Apr 2022 at 13:19, godfrey he  wrote:
> > >
> > > > Hi Konstantin & Chesnay,
> > > >
> > > > Thanks for driving this discussion, I am willing to volunteer as the
> > > > release manager for 1.16.
> > > >
> > > >
> > > > Best,
> > > > Godfrey
> > > >
> > > > Konstantin Knauf  于2022年4月26日周二 18:23写道：
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > With Flink 1.15 about to be released, the community has started
> > > planning
> > > > &
> > > > > developing features for the next release, Flink 1.16. As such, I
> would
> > > > like
> > > > > to start a discussion around managing this release.
> > > > >
> > > > > Specifically, Chesnay & myself would like to volunteer as release
> > > > managers.
> > > > > Our focus as release managers would be
> > > > > * to propose a release timeline
> > > > > * to provide an overview of all ongoing development threads and
> ideally
> > > > > their current status to the community
> > > > > * to keep an eye on build stability
> > > > > * facilitate release testing
> > > > > * to do the actual release incl. communication (blog post, etc.)
> > > > >
> > > > > Is anyone else interested in acting as a release manager for Flink
> > > 1.16?
> > > > If
> > > > > so, we are happy to make this a joint effort.
> > > > >
> > > > > Besides the question of who will act as a release manager, I
> think, we
> > > > can
> > > > > already use this thread to align on a timeline. For collecting
> features
> > > > and
> > > > > everything else, we would start a dedicated threads shortly.
> > > > >
> > > > > Given Flink 1.15 will be released in the next days, and aiming for
> a 4
> > > > > months release cycle including stabilization, this would mean
> *feature
> > > > > freeze at the end of July*. The exact date could be determined
> later.
> > > Any
> > > > > thoughts on the timeline.?
> > > > >
> > > > > Looking forward to your thoughts!
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Chesnay & Konstantin
> > > >
> > >
>

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

[Konstantin Knauf]

Hi Lijie,

I think, this makes sense and +1 to only support manually blocking
taskmanagers and nodes. Maybe the different strategies can also be
maintained outside of Apache Flink.

A few remarks:

1) Can we use another term than "bla.cklist" due to the controversy around
the term? [1] There was also a Jira Ticket about this topic a while back
and there was generally a consensus to avoid the term blacklist & whitelist
[2]? We could use "blocklist" "denylist" or "quarantined"
2) For the REST API, I'd prefer a slightly different design as verbs like
add/remove often considered an anti-pattern for REST APIs. POST on a list
item is generally the standard to add items. DELETE on the individual
resource is standard to remove an item.

POST /quarantine/items
DELETE /quarantine/items/

We could also consider to separate taskmanagers and nodes in the REST API
(and internal data structures). Any opinion on this?

POST/GET /quarantine/nodes
POST/GET /quarantine/taskmanager
DELETE /quarantine/nodes/
DELETE /quarantine/taskmanager/

3) How would blocking nodes behave with non-active resource managers, i.e.
standalone or reactive mode?

4) To keep the implementation even more minimal, do we need the timeout
behavior? If items are added/removed manually we could delegate this to the
user easily. In my opinion the timeout behavior would better fit into
specific strategies at a later point.

Looking forward to your thoughts.

Cheers and thank you,

Konstantin

[1]
https://en.wikipedia.org/wiki/Blacklist_(computing)#Controversy_over_use_of_the_term
[2] https://issues.apache.org/jira/browse/FLINK-18209

Am Mi., 27. Apr. 2022 um 04:04 Uhr schrieb Lijie Wang <
wangdachui9...@gmail.com>:

> Hi all,
>
> Flink job failures may happen due to cluster node issues (insufficient disk
> space, bad hardware, network abnormalities). Flink will take care of the
> failures and redeploy the tasks. However, due to data locality and limited
> resources, the new tasks are very likely to be redeployed to the same
> nodes, which will result in continuous task abnormalities and affect job
> progress.
>
> Currently, Flink users need to manually identify the problematic node and
> take it offline to solve this problem. But this approach has following
> disadvantages:
>
> 1. Taking a node offline can be a heavy process. Users may need to contact
> cluster administors to do this. The operation can even be dangerous and not
> allowed during some important business events.
>
> 2. Identifying and solving this kind of problems manually would be slow and
> a waste of human resources.
>
> To solve this problem, Zhu Zhu and I propose to introduce a blacklist
> mechanism for Flink to filter out problematic resources.
>
>
> You can find more details in FLIP-224[1]. Looking forward to your feedback.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-224%3A+Blacklist+Mechanism
>
>
> Best,
>
> Lijie
>

[DISCUSS] Planning Flink 1.16

[Konstantin Knauf]

Hi everyone,

With Flink 1.15 about to be released, the community has started planning &
developing features for the next release, Flink 1.16. As such, I would like
to start a discussion around managing this release.

Specifically, Chesnay & myself would like to volunteer as release managers.
Our focus as release managers would be
* to propose a release timeline
* to provide an overview of all ongoing development threads and ideally
their current status to the community
* to keep an eye on build stability
* facilitate release testing
* to do the actual release incl. communication (blog post, etc.)

Is anyone else interested in acting as a release manager for Flink 1.16? If
so, we are happy to make this a joint effort.

Besides the question of who will act as a release manager, I think, we can
already use this thread to align on a timeline. For collecting features and
everything else, we would start a dedicated threads shortly.

Given Flink 1.15 will be released in the next days, and aiming for a 4
months release cycle including stabilization, this would mean *feature
freeze at the end of July*. The exact date could be determined later. Any
thoughts on the timeline.?

Looking forward to your thoughts!

Thanks,

Chesnay & Konstantin

Re: [DISCUSS] Maintain a Calcite repository for Flink to accelerate the development for Flink SQL features

[Konstantin Knauf]

I share Chesay's concerns. I fear that we will generally upgrade Calcite
even fewer and - in consequence - when we upgrade it will be even more
painful.

Can you elaborate on why Calcite upgrades have been hard in the past? Is
there anything we can do to make this better?

On Fri, Apr 22, 2022 at 10:07 AM Chesnay Schepler 
wrote:

> I'm overall against the idea of creating a fork.
> It implies quite some maintenance overhead, like dealing with unstable
> tests, CI, licensing etc. and the overall release overhead.
>
> Is there no alternative where we can collaborate more with the calcite
> guys, like verifying new features so bugs are caught sooner?
>
> On 22/04/2022 09:31, godfrey he wrote:
> > Dear devs,
> >
> > I would like to open a discussion on the fact that currently many
> > Flink SQL function
> >   development relies on Calcite releases, which seriously blocks some
> > Flink SQL's features release.
> > Therefore, I would like to discuss whether it is possible to solve this
> problem
> > by creating Flink's own Calcite repository.
> >
> > Currently, Flink depends on Caclite-1.26, FLIP-204[1] relies on
> Calcite-1.30,
> > and we recently want to support fully join-hints functionatity in
> Flink-1.16,
> > which relies on Calcite-1.31 (maybe two or three months later will be
> released).
> >
> > In order to support some new features or fix some bugs, we need to
> upgrade
> > the Calcite version, but every time we upgrade Calcite version
> > (especially upgrades
> > across multiple versions), the processing is very tough: I remember
> clearly that
> >   the Calcite upgrade from 1.22 to 1.26 took two weeks of full-time to
> complete.
> >
> > Currently, in order to fix some bugs while not upgrading the Calcite
> version,
> > we copy the corresponding Calcite class directly into the Flink project
> > and then modify it accordingly.[2] This approach is rather hacky and
> > hard for code maintenance and upgrades.
> >
> > So, I had an idea whether we could solve this problem by maintaining a
> > Calcite repository
> > in the Flink community. This approach has been practiced within my
> > company for many years.
> >   There are similar practices in the industry. For example, Apache Dill
> > also maintains
> > a separate Calcite repository[3].
> >
> > The following is a brief analysis of the approach and the pros and
> > cons of maintaining a separate repository.
> >
> > Approach:
> > 1. Where to put the code? https://github.com/flink-extended is a good
> place.
> > 2. What extra code can be added to this repository? Only bug fixes and
> features
> > that are already merged into Calcite can be cherry-picked to this
> repository.
> > We also should try to push bug fixes to the Calcite community.
> > Btw, the copied Calcite class in the Flink project can be removed.
> > 3. How to upgrade the Calcite version? Check out the target Calcite
> > release branch
> > and rebase our bug fix code. (As we upgrade, we will maintain fewer
> > and fewer older bug
> > fixes code.) And then, verify all Calcte's tests and Flink's tests in
> > the developer's local
> >   environment. If all tests are OK, release the Calcite branch, or fix
> > it in the branch and re-test.
> >   After the branch is released, then the version of Calcite in Flink
> > can be upgraded. For example:
> >   checkout calcite-1.26.0-flink-v1-SNAPSHOT branch from calcite-1.26.0,
> > move all the copied
> >   Calcite code in Flink to the branch, and pick all the hint related
> > changes from Calcite-1.31 to
> >   the branch. Then we can change the Calcite version in Flink to
> > calcite-1.26.0-flink-v1-SNAPSHOT,
> > and verify all tests in the locale. Release calcite-1.26.0-flink-v1
> > after all tests are successful.
> > At last upgrade the calcite version to
> > calcite-1.26.0-flink-v10-flink-v1, and open a PR.
> > 4. Who will maintain it? The maintenance workload is minimal, but the
> > upgrade work is
> >   laborious (actually, it's similar to before). I can maintain it in
> > the early stage and standardise the processing.
> >
> > Pros.
> > 1. The release of Flink is decoupled from the release of Calcite,
> >   making feature development and bug fix quicker
> > 2. Reduce the hassle of unnecessary calcite upgrades
> > 3. No hacking in Flink to maintain the Calcite copied code
> >
> > cons.
> > 1. Need to maintain an additional Calcite repository
> > 2. The Upgrades are a little more complicated than before
> >
> > Any feedback is very welcome!
> >
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-204%3A+Introduce+Hash+Lookup+Join
> > [2]
> https://github.com/apache/flink/tree/master/flink-table/flink-table-planner/src/main/java/org/apache/calcite
> > [3] https://github.com/apache/drill/blob/master/pom.xml#L64
> >
> > Best,
> > Godfrey
>
>
>

-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

[Konstantin Knauf]

Hi Sebastian, Hi Dawid,

As part of this FLIP, the `AlignedSplitReader` interface (aka the stop &
resume behavior) will be implemented for Kafka and Pulsar only, correct?

+1 in general. I believe it is valuable to complete the watermark aligned
story with this FLIP.

Cheers,

Konstantin







On Thu, Apr 21, 2022 at 12:36 PM Dawid Wysakowicz 
wrote:

> To be explicit, having worked on it, I support it ;) I think we can
> start a vote thread soonish, as there are no concerns so far.
>
> Best,
>
> Dawid
>
> On 13/04/2022 11:27, Sebastian Mattheis wrote:
> > Dear Flink developers,
> >
> > I would like to open a discussion on FLIP 217 [1] for an extension of
> > Watermark Alignment to perform alignment also in SplitReaders. To do so,
> > SplitReaders must be able to suspend and resume reading from split
> sources
> > where the SourceOperator coordinates and controlls suspend and resume. To
> > gather information about current watermarks of the SplitReaders, we
> extend
> > the internal WatermarkOutputMulitplexer and report watermarks to the
> > SourceOperator.
> >
> > There is a PoC for this FLIP [2], prototyped by Arvid Heise and revised
> and
> > reworked by Dawid Wysakowicz (He did most of the work.) and me. The
> changes
> > are backwards compatible in a way that if affected components do not
> > support split alignment the behavior is as before.
> >
> > Best,
> > Sebastian
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-217+Support+watermark+alignment+of+source+splits
> > [2] https://github.com/dawidwys/flink/tree/aligned-splits
> >
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: Alertmanager Sink Connector

[Konstantin Knauf]

cc user@, bcc dev@

Hi Dhruv,

Yes, this should be totally possible.

For 1, I would use a ProcessFunction to buffer the alerts and register
timers per alert for the repeated firing (or to clear it from state if it
is resolved). From this ProcessFunction you send records to an
AlertManagerSink.

For 2. the AsyncSink, Fabian Paul (cc) might be the best person to judge if
it is a good fit. There is PR [1] to add a blog post about the Async Sink
that might be of help for you in the meantime.

Cheers,

Konstantin

[1] https://github.com/apache/flink-web/pull/517


On Thu, Apr 14, 2022 at 9:43 AM Dhruv Patel 
wrote:

> Hi,
>We have a use case where we want to send alerts to Prometheus
> Alertmanager (https://prometheus.io/docs/alerting/latest/alertmanager/)
> from Flink. Each of these alerts have a startsAt, endsAt field along with
> alertMetadata. Alertmanager expects clients to send all the FIRING alerts
> every X minutes (X is configurable) with an updated endsAt time (greater
> than X since alertmanager would drop the alert once endsAt time is reached)
> and once the alert is in RESOLVED state stop sending it. The state updates
> of the alerts would come from Kafka. So the pipeline is something like this
> State Updates to Alerts (Kafka) -> Flink (Some Enrichment) -> Alertmanager
> Sink
>  They provide a rest endpoint where we can post these alerts. I have some
> questions to see if its achievable to develop a sink connector for
> alertmanager in flink?
>
> 1. Is it possible to send data to a sink every X minutes from a custom sink
> connector since I have to somehow simulate a behavior of continuously
> sending the same alerts even because state updates are only received from
> Kafka for FIRING -> RESOLVED state and not for FIRING -> FIRING state? I
> was thinking of having a state of active alerts and somehow the sink
> connector would get the state every X minutes, batch it and then send it to
> alertmanager. However, I am not able to find resources to write some
> implementation around it.
>
> 2. In Flink 1.15, there are Async Sinks (
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink)
> but not much documentation around. Also don't know if it would be
> achievable to write the continuous firing logic in alertmanager
>
> Any other ideas are welcomed.
>
> --
> *Thanks,*
> *Dhruv*
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: flink提交错误时报错信息不清晰

[Konstantin Knauf]

Hi there,

Please use English on the dev@flink.apache.org mailing list.
Images/attachements don't work on these mailing lists. You would have to
host it somewhere else and link to it.

If this was a question about using Apache Flink user@ (Englisch) or user-zh@
(Chinese) would be the correct mailing lists to address.

Cheers,

Konstantin



On Thu, Mar 24, 2022 at 11:48 AM 丛鹏  wrote:

> 嗨，我这边在使用flink1.12.2提交yarn-application的任务时发现一个问题，
> 提交任务的flink官网的例子为
> ./bin/flink run-application -t yarn-application
> ./examples/streaming/TopSpeedWindowing.jar
>
> 如果其中有部分写错的话 yarn-application 写成了 yarn-appliation的话
> 就会报错
> [image: image.png]
> 这里我看了代码是CliFrontend 封装的 213行的配置集合 effectiveConfiguration
> 有问题，导致DefaultClusterClientServiceLoader.java:83 判断进入报错
> [image: image.png]
> 我觉得这里的报错信息的描述有问题 会造成误导，是使用者误认为是自己的HADOOP_CLASSPATH 环境有问题，希望可以得到大家的回复
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [DISCUSS] Structure of the Flink Documentation (Languages & APIs)

[Konstantin Knauf]

Hi Dian,

Thank you for sharing your thoughts. What do you propose going forward? I
am not sure I got this from your email.

Best,

Konstantin



On Wed, Mar 23, 2022 at 10:03 AM Dian Fu  wrote:

> Hi Konstantin,
>
> Thanks a lot for bringing up this discussion.
>
> Currently, the Python documentation is more like a mixture of Option 1 and
> Option 2. It contains two parts:
> 1) The first part is the independent page [1] which could be seen as the
> main entrypoint for Python users.
> 2) The second part is the Python tabs which are among the DataStream API /
> Table API pages.
>
> The motivation to provide an independent page for Python documentation is
> as follows:
> 1) We are trying to create a Pythonic documentation for Python users (we
> are still far away from that and I have received much feedback saying that
> the Python documentation and API is too Java-like). However, to avoid
> duplication, it will link to the DataStream API / Table API pages when
> necessary instead of copying content. There are indeed exceptions, e.g. the
> window example given by Jark, that's because it only provides a very
> limited window support in Python DataStream API at present and to give
> Python users a complete picture of what they can do in Python DataStream
> API, we have added a dedicated page. We are trying to finalize the window
> support in 1.16 [2] and remove the duplicate documentation.
> 2) There are some kinds of documentations which are only applicable for
> Python language, e.g. dependency management[2], conversion between Table
> and Pandas DataFrame [3], etc. Providing an independent page helps to
> provide a place to hold all these kinds of documentation together.
>
> Regarding Option 1: "Language Tabs", this makes it hard to create Pythonic
> documentation for Python users.
> Regarding Option 2: "Language First", it may mean a lot of duplications.
> Currently, there are a lot of descriptions in the DataStream API / Table
> API pages which are shared between Java/Scala/Python.
>
> > In the rest of the documentation, Python is sometimes
> > included like in this Table API page [2] and sometimes ignored like on
> the
> > project setup pages [3].
> I agree that this is something that we need to improve.
>
> Regards,
> Dian
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/overview/
> [2] https://issues.apache.org/jira/browse/FLINK-26477
> [2]
>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/dependency_management/
> [3]
>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/table/conversion_of_pandas/
>
> On Wed, Mar 23, 2022 at 4:17 PM Jark Wu  wrote:
>
> > Hi Konstantin,
> >
> > Thanks for starting this discussion.
> >
> > From my perspective, I prefer the "Language Tabs" approach.
> > But maybe we can improve the tabs to move to the sidebar or top menu,
> > which allows users to first decide on their language and then the API.
> > IMO, programming languages are just like spoken languages which can be
> > picked in the sidebar.
> > What I want to avoid is the duplicate docs and in-complete features in a
> > specific language.
> > "Language First" may confuse users about what is and where to find the
> > complete features provided by flink.
> >
> > For example, there are a lot of duplications in the "Window" pages[1] and
> > "Python Window" pages[2].
> > And users can't have a complete overview of Flink's window mechanism from
> > the Python API part.
> > Users have to go through the Java/Scala DataStream API first to build the
> > overall knowledge,
> > and then to read the Python API part.
> >
> > > * Second, most of the Flink Documentation currently is using a
> "Language
> > Tabs" approach, but this might become obsolete in the long-term anyway as
> > we move more and more in a Scala-free direction.
> >
> > The Scala-free direction means users can pick arbitrary Scala versions,
> not
> > drop the Scala API.
> > So the "Language Tabs" is still necessary and helpful for switching
> > languages.
> >
> > Best,
> > Jark
> >
> > [1]:
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/datastream/operators/windows/
> > [2]:
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/
> >
> >
> >
> >
> >
> >
> >
> > On Tue, 22 Mar 2022 at 21:40, Konstantin Knauf 
> wrote:
> >
> > > Hi everyone,
> &g

Re: [DISCUSS] Structure of the Flink Documentation (Languages & APIs)

[Konstantin Knauf]

Hi Jark,

Thank you for joining the discussion. I am totally fine with the "Language
Tabs". Would be good to hear what the PyFlink community thinks, because the
Python Documentation is the biggest exception to the "Language Tabs"
approach.

@Jark Wu : Do you agree that SQL should be separate from
Table API/DataStream API in either way? I think SQL does neither fit into
the "Language Tabs" nor "Language First".

Cheers,

Konstantin


On Wed, Mar 23, 2022 at 9:17 AM Jark Wu  wrote:

> Hi Konstantin,
>
> Thanks for starting this discussion.
>
> From my perspective, I prefer the "Language Tabs" approach.
> But maybe we can improve the tabs to move to the sidebar or top menu,
> which allows users to first decide on their language and then the API.
> IMO, programming languages are just like spoken languages which can be
> picked in the sidebar.
> What I want to avoid is the duplicate docs and in-complete features in a
> specific language.
> "Language First" may confuse users about what is and where to find the
> complete features provided by flink.
>
> For example, there are a lot of duplications in the "Window" pages[1] and
> "Python Window" pages[2].
> And users can't have a complete overview of Flink's window mechanism from
> the Python API part.
> Users have to go through the Java/Scala DataStream API first to build the
> overall knowledge,
> and then to read the Python API part.
>
> > * Second, most of the Flink Documentation currently is using a "Language
> Tabs" approach, but this might become obsolete in the long-term anyway as
> we move more and more in a Scala-free direction.
>
> The Scala-free direction means users can pick arbitrary Scala versions, not
> drop the Scala API.
> So the "Language Tabs" is still necessary and helpful for switching
> languages.
>
> Best,
> Jark
>
> [1]:
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/datastream/operators/windows/
> [2]:
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/
>
>
>
>
>
>
>
> On Tue, 22 Mar 2022 at 21:40, Konstantin Knauf  wrote:
>
> > Hi everyone,
> >
> > I would like to discuss a particular aspect of our documentation: the
> > top-level structure with respect to languages and APIs. The current
> > structure is inconsistent and the direction is unclear to me, which makes
> > it hard for me to contribute gradual improvements.
> >
> > Currently, the Python documentation has its own independent branch in the
> > documentation [1]. In the rest of the documentation, Python is sometimes
> > included like in this Table API page [2] and sometimes ignored like on
> the
> > project setup pages [3]. Scala and Java on the other hand are always
> > documented in parallel next to each other in tabs.
> >
> > The way I see it, most parts (application development, connectors,
> getting
> > started, project setup) of our documentation have two primary dimensions:
> > API (DataStream, Table API), Language (Python, Java, Scala)
> >
> > In addition, there is SQL, for which the language is only a minor factor
> > (UDFs), but which generally requires a different structure (different
> > audience, different tools). On the other hand, SQL and Table API have
> some
> > conceptual overlap, whereas I doubt these concepts are of big interest
> > to SQL users. So, to me SQL should be treated separately in any case with
> > links to the Table API documentation for some concepts.
> >
> > I think, in general, both approaches can work:
> >
> >
> > *Option 1: "Language Tabs"*
> > Application Development
> > > DataStream API  (Java, Scala, Python)
> > > Table API (Java, Scala, Python)
> > > SQL
> >
> >
> > *Option 2: "Language First" *
> > Java Development Guide
> > > Getting Started
> > > DataStream API
> > > Table API
> > Python Development Guide
> > > Getting Started
> > > Datastream API
> > > Table API
> > SQL Development Guide
> >
> > I don't have a strong opinion on this, but tend towards "Language First".
> >
> > * First, I assume, users actually first decide on their language/tools of
> > choice and then move on to the API.
> >
> > * Second, most of the Flink Documentation currently is using a "Language
> > Tabs" approach, but this might become obsolete in the long-term anyway as
> > we move more and more in a Scala-free direction.
> >
> > For the conne

[DISCUSS] Structure of the Flink Documentation (Languages & APIs)

[Konstantin Knauf]

Hi everyone,

I would like to discuss a particular aspect of our documentation: the
top-level structure with respect to languages and APIs. The current
structure is inconsistent and the direction is unclear to me, which makes
it hard for me to contribute gradual improvements.

Currently, the Python documentation has its own independent branch in the
documentation [1]. In the rest of the documentation, Python is sometimes
included like in this Table API page [2] and sometimes ignored like on the
project setup pages [3]. Scala and Java on the other hand are always
documented in parallel next to each other in tabs.

The way I see it, most parts (application development, connectors, getting
started, project setup) of our documentation have two primary dimensions:
API (DataStream, Table API), Language (Python, Java, Scala)

In addition, there is SQL, for which the language is only a minor factor
(UDFs), but which generally requires a different structure (different
audience, different tools). On the other hand, SQL and Table API have some
conceptual overlap, whereas I doubt these concepts are of big interest
to SQL users. So, to me SQL should be treated separately in any case with
links to the Table API documentation for some concepts.

I think, in general, both approaches can work:


*Option 1: "Language Tabs"*
Application Development
> DataStream API  (Java, Scala, Python)
> Table API (Java, Scala, Python)
> SQL


*Option 2: "Language First" *
Java Development Guide
> Getting Started
> DataStream API
> Table API
Python Development Guide
> Getting Started
> Datastream API
> Table API
SQL Development Guide

I don't have a strong opinion on this, but tend towards "Language First".

* First, I assume, users actually first decide on their language/tools of
choice and then move on to the API.

* Second, most of the Flink Documentation currently is using a "Language
Tabs" approach, but this might become obsolete in the long-term anyway as
we move more and more in a Scala-free direction.

For the connectors, I think, there is a good argument for "Language & API
Embedded", because documenting every connector for each API and language
separately would result in a lot of duplication. Here, I would go one step
further then what we have right now and target

Connectors
-> Kafka (All APIs incl. SQL, All Languages)
-> Kinesis (same)
-> ...

This also results in a quick overview for users about which connectors
exist and plays well with our plan of externalizing connectors.

For completeness & scope of the discussion: there are two outdated FLIPs on
documentation (42, 60), which both have not been implemented, are partially
contradicting each other and are generally out-of-date. I specifically
don't intend to add another FLIP to this graveyard, but still reach a
consensus on the high-level direction.

What do you think?

Cheers,

Konstantin

-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [ANNOUNCE] Apache Flink 1.1.4.4 released

[Konstantin Knauf]

Hi Xingbo,

you are totally right. Thank you for noticing. This also affected Flink
1.13.6, the other release I was recently managing. I simply skipped a step
in the release guide.

It should be fixed now. Could you double-check?

Cheers,

Konstantin

On Wed, Mar 16, 2022 at 4:07 AM Xingbo Huang  wrote:

> Thanks a lot for being our release manager Konstantin and everyone who
> contributed. I have a question about pyflink. I see that there are no
> corresponding wheel packages uploaded on pypi, only the source package is
> uploaded. Is there something wrong with building the wheel packages?
>
> Best,
> Xingbo
>
> Leonard Xu  于2022年3月16日周三 01:02写道：
>
>> Thanks a lot for being our release manager Konstantin and everyone who
>> involved!
>>
>> Best,
>> Leonard
>>
>> 2022年3月15日 下午9:34，Martijn Visser  写道：
>>
>> Thank you Konstantin and everyone who contributed!
>>
>>
>>

-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

[ANNOUNCE] Apache Flink 1.1.4.4 released

[Konstantin Knauf]

The Apache Flink community is very happy to announce the release of Apache
Flink 1.14.4, which is the third bugfix release for the Apache Flink 1.14
series.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

Please check out the release blog post for an overview of the improvements
for this bugfix release:

https://flink.apache.org/news/2022/03/11/release-1.14.4.html

The full release notes are available in Jira:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351231

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,

Konstantin

--

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [ANNOUNCE] New PMC member: Yuan Mei

[Konstantin Knauf]

Congratulations, Yuan!

On Mon, Mar 14, 2022 at 11:29 AM Jing Zhang  wrote:

> Congratulations, Yuan!
>
> Best,
> Jing Zhang
>
> Jing Ge  于2022年3月14日周一 18:15写道：
>
> > Congrats! Very well deserved!
> >
> > Best,
> > Jing
> >
> > On Mon, Mar 14, 2022 at 10:34 AM Piotr Nowojski 
> > wrote:
> >
> > > Congratulations :)
> > >
> > > pon., 14 mar 2022 o 09:59 Yun Tang  napisał(a):
> > >
> > > > Congratulations, Yuan!
> > > >
> > > > Best,
> > > > Yun Tang
> > > > 
> > > > From: Zakelly Lan 
> > > > Sent: Monday, March 14, 2022 16:55
> > > > To: dev@flink.apache.org 
> > > > Subject: Re: [ANNOUNCE] New PMC member: Yuan Mei
> > > >
> > > > Congratulations, Yuan!
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Mon, Mar 14, 2022 at 4:49 PM Johannes Moser 
> > > wrote:
> > > >
> > > > > Congrats Yuan.
> > > > >
> > > > > > On 14.03.2022, at 09:45, Arvid Heise  wrote:
> > > > > >
> > > > > > Congratulations and well deserved!
> > > > > >
> > > > > > On Mon, Mar 14, 2022 at 9:30 AM Matthias Pohl  >
> > > > wrote:
> > > > > >
> > > > > >> Congratulations, Yuan.
> > > > > >>
> > > > > >> On Mon, Mar 14, 2022 at 9:25 AM Shuo Cheng 
> > > > wrote:
> > > > > >>
> > > > > >>> Congratulations, Yuan!
> > > > > >>>
> > > > > >>> On Mon, Mar 14, 2022 at 4:22 PM Anton Kalashnikov <
> > > > kaa@yandex.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Congratulations, Yuan!
> > > > > >>>>
> > > > > >>>> --
> > > > > >>>>
> > > > > >>>> Best regards,
> > > > > >>>> Anton Kalashnikov
> > > > > >>>>
> > > > > >>>> 14.03.2022 09:13, Leonard Xu пишет:
> > > > > >>>>> Congratulations Yuan!
> > > > > >>>>>
> > > > > >>>>> Best,
> > > > > >>>>> Leonard
> > > > > >>>>>
> > > > > >>>>>> 2022年3月14日 下午4:09，Yangze Guo  写道：
> > > > > >>>>>>
> > > > > >>>>>> Congratulations!
> > > > > >>>>>>
> > > > > >>>>>> Best,
> > > > > >>>>>> Yangze Guo
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Mar 14, 2022 at 4:08 PM Martijn Visser <
> > > > > >>>> martijnvis...@apache.org> wrote:
> > > > > >>>>>>> Congratulations Yuan!
> > > > > >>>>>>>
> > > > > >>>>>>> On Mon, 14 Mar 2022 at 09:02, Yu Li 
> > wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> Hi all!
> > > > > >>>>>>>>
> > > > > >>>>>>>> I'm very happy to announce that Yuan Mei has joined the
> > Flink
> > > > PMC!
> > > > > >>>>>>>>
> > > > > >>>>>>>> Yuan is helping the community a lot with creating and
> > > validating
> > > > > >>>> releases,
> > > > > >>>>>>>> contributing to FLIP discussions and good code
> contributions
> > > to
> > > > > >> the
> > > > > >>>>>>>> state backend and related components.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Congratulations and welcome, Yuan!
> > > > > >>>>>>>>
> > > > > >>>>>>>> Best Regards,
> > > > > >>>>>>>> Yu (On behalf of the Apache Flink PMC)
> > > > > >>>>>>>>
> > > > > >>>> --
> > > > > >>>>
> > > > > >>>> Best regards,
> > > > > >>>> Anton Kalashnikov
> > > > > >>>>
> > > > > >>>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [DISCUSS] Preview release for Flink Kubernetes Operator

[Konstantin Knauf]

Hi everyone,

can we mark all the APIs as experimental/alpha so that it is clear that
these can be broken in future releases for now? I think this would be very
important given the early stage of the project. We want to be able to
address shortcomings without worrying too much about backwards
compatibility at this stage, I believe.

Cheers,

Konstantin

On Sun, Mar 13, 2022 at 7:48 AM Yang Wang  wrote:

> Thanks Gyula for starting this discussion.
>
> Given that the core functionality is closing to stable, I am in favor of
> having the MVP release at the end of March.
> The first release will help us to collect more feedbacks from the users.
> Also it is a good chance to let the users know that the community is trying
> to maintain an official Kubernetes operator :)
> I hope that the companies could build their own production streaming
> platform on top of the flink-kubernetes-operator in the future.
>
> FYI: @Wenjun Min is still working hard on supporting the Session Job in
> Flink Kubernetes operator, It will be great if we could include it in the
> first release.
> And I believe we have enough time.
>
> Moreover, I agree with you that we need to invest more time in the
> documentation, e2e tests, helm install optimization, logging,
> etc. before the release.
>
>
> Best,
> Yang
>
>
> Gyula Fóra  于2022年3月12日周六 01:10写道：
>
> > Hi Team!
> >
> > I would like to discuss the timeline for the initial preview/milestone
> > release of the flink-kubernetes-operator
> > <https://github.com/apache/flink-kubernetes-operator> project.
> >
> > The last few weeks we have been working very hard with the community to
> > stabilize the initial feature set and I think we have made great
> progress.
> > While we are still far from a production ready-state, a preview release
> > will give us the opportunity to reach more people and gather much needed
> > input to take this project to the next level.
> >
> > There are still a couple missing features that we need to iron out and we
> > need to make sure we have proper documentation but after that I think it
> > would be a good time for the preview release.
> >
> > I propose to aim for the first release candidate around the 25-27th of
> > March after which we should dedicate a few days for some extensive
> testing
> > and bugfixing.
> >
> > What do you think?
> >
> > Gyula
> >
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: Add libatomic1 to flink image

[Konstantin Knauf]

Moving dev@ to bcc, adding user@

Hi Julius,

the recommended approach would be to build your own Docker images from the
official images along the lines of

FROM apache/flink:1.14.3
RUN apt-get install -y libatomic1

Cheers,

Konstantin


On Fri, Mar 11, 2022 at 11:07 AM Almeida, Julius
 wrote:

> Hi Developers,
>
> I wish to add a new library to existing flink docker image since it’s a
> dependency requirement in my processor.
> Is it possible to add?
>
> apt-get install libatomic1
>
> Thanks,
> Julius
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk