Re: [SUMMARY] Flink 1.19 Release Sync 11/14/2023

[Matthias Pohl]

Thanks for the update, Lincoln.
A small remark: The appreciation for fixing FLINK-18356 [1] should be sent
to Yunhong Zheng. :)

Now, let's see and hope that the problem doesn't appear again.

Best,
Matthias

[1] https://issues.apache.org/jira/browse/FLINK-18356

On Wed, Nov 29, 2023 at 2:41 AM Lincoln Lee  wrote:

> Hi devs,
>
> This is the summary of the second release sync of Flink 1.19:
>
> - Features & issues tracking
> There're 9 weeks until the feature freeze date, so far we've had 21 flips
> and the status looks good. It is encouraged to continuously updating the
> 1.19 wiki page[1] for contributors.
>   - GitHub Actions Migration is under discussion (FLIP-396[2]), volunteers
> are welcome!
>   - The flink-shaded[3] version 1.18.0 will be released soon, thanks
> Sergey for driving this!
>
> - Blockers
>   - [new] external connectors compilation failure cause by FLINK-25857
> , now back to the ml
> discussion[4]
>   - [not a blocker] FLINK-31449 Remove DeclarativeSlotManager related
> logic - changed to major
>   - [closed] FLINK-33531 Nightly Python fails - thanks Xingbo & Dian for
> fixing this
>   - [closed] FLINK-18356 flink-table-planner Exit code 137 on ci pipeline
> - thanks @Matthias for getting it done
>
> - Sync meeting
> The Google Meet has been updated to: https://meet.google.com/vcx-arzs-trv.
> As suggested by Xintong, we encourage attendees to fill out the topics to
> be discussed at the bottom of 1.19 wiki page[1] a day in advance to make it
> easier for everyone to understand the background of the topic.
>
> The next release sync will be on December 12th, 2023.
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/1.19+Release
> [2] https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb
> [3] https://lists.apache.org/thread/gkly9to8vnx9wrp7tvfl39s6tm4xdc9p
> [4] https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57
>
> Best regards,
> Yun, Jing, Martijn and Lincoln
>

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Matthias Pohl]

>
> According to the Flip, the new tests will support arm env.
> I believe that's good news for arm users. I have a minor
> question here. Will it be a blocker before migrating the new
> tests? If not,  If not, when can we expect arm environment
> support to be implemented? Thanks.


Thanks for your feedback, everyone.

About the ARM support. I want to underline that this FLIP is not about
migrating to GitHub Actions but to start a trial run in the Apache Flink
repository. That would allow us to come up with a proper decision whether
GitHub Actions is what we want. I admit that the title is a bit
"clickbaity". I updated the FLIP's title and its Motivation to make things
clear.

The FLIP suggests starting a trial period until 1.19 is released to try
things out. A proper decision on whether we want to migrate would be made
at the end of the 1.19 release cycle.

About the ARM support: This related content of the FLIP is entirely based
on documentation from Apache INFRAs side. INFRA seems to offer this ARM
support for their ephemeral runners. The ephemeral runners are in the
testing stage, i.e. these runners are still experimental. Apache INFRA asks
Apache projects to join this test.

Whether the ARM support is actually possible to achieve within Flink is
something we have to figure out as part of the trial run. One conclusion of
the trial run could be that we still move ahead with GHA but don't use arm
machines due to some blocking issues.

Matthias



On Wed, Nov 29, 2023 at 4:46 AM Yuxin Tan  wrote:

> Hi, Matthias,
>
> Thanks for driving this.
> +1 from my side.
>
> According to the Flip, the new tests will support arm env.
> I believe that's good news for arm users. I have a minor
> question here. Will it be a blocker before migrating the new
> tests? If not,  If not, when can we expect arm environment
> support to be implemented? Thanks.
>
> Best,
> Yuxin
>
>
> Márton Balassi  于2023年11月29日周三 03:09写道：
>
> > Thanks, Matthias. Big +1 from me.
> >
> > On Tue, Nov 28, 2023 at 5:30 PM Matthias Pohl
> >  wrote:
> >
> > > Thanks for the pointer. I'm planning to join that meeting.
> > >
> > > On Tue, Nov 28, 2023 at 4:16 PM Etienne Chauchot  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > FYI there is the ASF infra roundtable soon. One of the subjects for
> > this
> > > > session is GitHub Actions. It could be worth passing by:
> > > >
> > > > December 6th, 2023 at 1700 UTC on the #Roundtablechannel on Slack.
> > > >
> > > > For information about theroundtables, and about how to join,
> > > > see:https://infra.apache.org/roundtable.html
> > > > 
> > > >
> > > > Best
> > > >
> > > > Etienne
> > > >
> > > > Le 24/11/2023 à 14:16, Maximilian Michels a écrit :
> > > > > Thanks for reviving the efforts here Matthias! +1 for the
> transition
> > > > > to GitHub Actions.
> > > > >
> > > > > As for ASF Infra Jenkins, it works fine. Jenkins is extremely
> > > > > feature-rich. Not sure about the spare capacity though. I know that
> > > > > for Apache Beam, Google donated a bunch of servers to get
> additional
> > > > > build capacity.
> > > > >
> > > > > -Max
> > > > >
> > > > >
> > > > > On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl
> > > > >   wrote:
> > > > >> Btw. even though we've been focusing on GitHub Actions with this
> > FLIP,
> > > > I'm
> > > > >> curious whether somebody has experience with Apache Infra's
> Jenkins
> > > > >> deployment. The discussion I found about Jenkins [1] is quite
> > > out-dated
> > > > >> (2014). I haven't worked with it myself but could imagine that
> there
> > > are
> > > > >> some features provided through plugins which are missing in GitHub
> > > > Actions.
> > > > >>
> > > > >> [1]
> https://lists.apache.org/thread/vs81xdhn3q777r7x9k7wd4dyl9kvoqn4
> > > > >>
> > > > >> On Tue, Nov 21, 2023 at 4:19 PM Matthias Pohl<
> > matthias.p...@aiven.io>
> > > > >> wrote:
> > > > >>
> > > > >>> That's a valid point. I updated the FLIP accordingly:
> > > > >>>
> > > >  Currently, the secrets (e.g. for S3 access tokens) are
> maintained
> > by
> > > >  certain PMC members with access to the corresponding
> configuration
> > > in
> > > > the
> > > >  Azure CI project. This responsibility will be moved to Apache
> > Infra.
> > > > They
> > > >  are in charge of handling secrets in the Apache organization.
> As a
> > > >  consequence, updating secrets is becoming a bit more
> complicated.
> > > > This can
> > > >  be still considered an improvement from a legal standpoint
> because
> > > the
> > > >  responsibility is transferred from an individual company (i.e.
> > > > Ververica
> > > >  who's the maintainer of the Azure CI project) to the Apache
> > > > Foundation.
> > > > >>>
> > > > >>> On Tue, Nov 21, 2023 at 3:37 PM Martijn Visser<
> > > > martijnvis...@apache.org>
> > > > >>> wrote:
> > > > >>>
> > > >  Hi Matthias,
> > > > 
> > > >  Thanks for the write-up and for the efforts on this. I really
> 

[jira] [Created] (FLINK-33690) build_wheels_on_macos sometimes fails with read timed out on AZP

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-33690:
---

 Summary: build_wheels_on_macos sometimes fails with read timed out 
on AZP
 Key: FLINK-33690
 URL: https://issues.apache.org/jira/browse/FLINK-33690
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.18.0
Reporter: Sergey Nuyanzin


This build 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55013=logs=f73b5736-8355-5390-ec71-4dfdec0ce6c5=90f7230e-bf5a-531b-8566-ad48d3e03bbb=217
failed as
{noformat}
...
File 
"/private/var/folders/3s/vfzpb5r51gs6y328rmlgzm7cgn/T/pip-standalone-pip-3jt0u_hl/__env_pip__.zip/pip/_vendor/urllib3/response.py",
 line 573, in stream
  data = self.read(amt=amt, decode_content=decode_content)
File 
"/private/var/folders/3s/vfzpb5r51gs6y328rmlgzm7cgn/T/pip-standalone-pip-3jt0u_hl/__env_pip__.zip/pip/_vendor/urllib3/response.py",
 line 538, in read
  raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File 
"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py",
 line 131, in __exit__
  self.gen.throw(type, value, traceback)
File 
"/private/var/folders/3s/vfzpb5r51gs6y328rmlgzm7cgn/T/pip-standalone-pip-3jt0u_hl/__env_pip__.zip/pip/_vendor/urllib3/response.py",
 line 440, in _error_catcher
  raise ReadTimeoutError(self._pool, None, "Read timed out.")
  pip._vendor.urllib3.exceptions.ReadTimeoutError: 
HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSSION] flink-connector-shared-utils release process

[Leonard Xu]

Thanks Etienne for update the wiki, 

I just checked the change by comparing with history version, LGTM


Best,
Leonard

> 2023年11月28日 下午10:43，Etienne Chauchot  写道：
> 
> Hi all,
> 
> I just updated the doc: 
> https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development
> 
> Best
> 
> Etienne
> 
> Le 27/11/2023 à 12:43, Etienne Chauchot a écrit :
>> 
>> Sure!
>> 
>> Le 23/11/2023 à 02:57, Leonard Xu a écrit :
>>> Thanks Etienne for driving this.
>>> 
 - a flink-connector-shared-utils-*test* clone repo and a 
 *io.github.user.flink*:flink-connector-parent custom artifact to be able 
 to directly commit and install the artifact in the CI
 - a custom ci script that does the cloning and mvn install in the ci.yml 
 github action script for testing with the new flink-connector-parent 
 artifact
 If people agree on the process and location
>>> +1 for the process and location, could we also add a short paragraph and 
>>> the location link as well in [1] to remind connector developers?
>>> 
>>> Best,
>>> Leonard
>>> 
>>> [1]https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development
>>> 
>>> 

Re: [ANNOUNCE] Apache Flink 1.17.2 released

[Leonard Xu]

Thanks Yun for driving the release.  
Thanks a lot to everyone that has contributed with bug fixes and other 
improvements!

Best,
Leonard


> 2023年11月29日 下午1:05，Yun Tang  写道：
> 
> The Apache Flink community is very happy to announce the release of Apache 
> Flink 1.17.2, which is the second bugfix release for the Apache Flink 1.17 
> series.
> 
> Apache Flink® Is a framework and distributed processing engine for stateful 
> computations over unbounded and bounded data streams. Flink has been designed 
> to run in all common cluster environments, perform computations at in-memory 
> speed and at any scale.
> 
> The release is available for download at:
> https://flink.apache.org/downloads.html 
> 
> 
> Please check out the release blog post for an overview of the improvements 
> for this bugfix release: 
> https://flink.apache.org/2023/11/29/apache-flink-1.17.2-release-announcement/ 
> 
> 
> The full release notes are available in Jira:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353260
>  
> 
> 
> We would like to thank all contributors of the Apache Flink community who 
> made this release possible!
> 
> 
> Feel free to reach out to the release managers (or respond to this thread) 
> with feedback on the release process. Our goal is to constantly improve the 
> release process. Feedback on what could be improved or things that didn't go 
> so well are appreciated.
> 
> 
> Regards,
> Release Manager

[jira] [Created] (FLINK-33689) jsonObjectAggFunction can't retract previous data which is invalid when enable local global agg

[Shuai Xu (Jira)]

Shuai Xu created FLINK-33689:


 Summary: jsonObjectAggFunction can't retract previous data which 
is invalid when enable local global agg
 Key: FLINK-33689
 URL: https://issues.apache.org/jira/browse/FLINK-33689
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Affects Versions: 1.18.0
Reporter: Shuai Xu


Run the test as following and enable LocalGlobal and minibatch  in 
sql/AggregateITCase . 
{code:java}
//代码占位符
def testGroupJsonObjectAggWithRetract(): Unit = {
  val data = new mutable.MutableList[(Long, String, Long)]
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  data.+=((2L, "Hallo", 2L))
  val sql =
s"""
   |select
   |   JSON_OBJECTAGG(key k value v)
   |from (select
   | cast(SUM(a) as string) as k,t as v
   |   from
   | Table6
   |   group by t)
   |""".stripMargin
  val t = failingDataSource(data).toTable(tEnv, 'a, 'c, 't)
  tEnv.createTemporaryView("Table6", t)
  val sink = new TestingRetractSink
  tEnv.sqlQuery(sql).toRetractStream[Row].addSink(sink).setParallelism(1)
  env.execute()
  val expected =
List(
  "{\"30\":2}"
)
  assertThat(sink.getRetractResults).isEqualTo(expected)
} {code}
The result is as following.
{code:java}
//代码占位符
List({"14":2,"30":2}) {code}
However,  \{"14":2}  should be retracted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33688) Reuse Channels in RestClient to save connection establish time

[xiangyu feng (Jira)]

xiangyu feng created FLINK-33688:


 Summary: Reuse Channels in RestClient to save connection establish 
time
 Key: FLINK-33688
 URL: https://issues.apache.org/jira/browse/FLINK-33688
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Reporter: xiangyu feng


RestClient can reuse the connections to Dispatcher when submitting http 
requests to a long running Flink cluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33687) Reuse RestClusterClient in StandaloneClusterDescriptor to avoid frequent thread create/destroy

[xiangyu feng (Jira)]

xiangyu feng created FLINK-33687:


 Summary: Reuse RestClusterClient in StandaloneClusterDescriptor to 
avoid frequent thread create/destroy
 Key: FLINK-33687
 URL: https://issues.apache.org/jira/browse/FLINK-33687
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Reporter: xiangyu feng


`RestClusterClient` can also be reused when submitting programs to a 
long-running Flink Cluster



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33686) Reuse StandaloneClusterDescriptor in RemoteExecutor when executing jobs on the same cluster

[xiangyu feng (Jira)]

xiangyu feng created FLINK-33686:


 Summary: Reuse StandaloneClusterDescriptor in RemoteExecutor when 
executing jobs on the same cluster
 Key: FLINK-33686
 URL: https://issues.apache.org/jira/browse/FLINK-33686
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Reporter: xiangyu feng


Multiple `RemoteExecutor` instances can reuse the same 
`StandaloneClusterDescriptor` when executing jobs to a same running cluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33685) StandaloneClusterId need to distinguish different remote clusters

[xiangyu feng (Jira)]

xiangyu feng created FLINK-33685:


 Summary: StandaloneClusterId need to distinguish different remote 
clusters
 Key: FLINK-33685
 URL: https://issues.apache.org/jira/browse/FLINK-33685
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Reporter: xiangyu feng


`StandaloneClusterId` is a singleton, which means `StandaloneClusterDescriptor` 
cannot distinguish different remote running clusters.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33684) Improve the retry strategy in CollectResultFetcher

[xiangyu feng (Jira)]

xiangyu feng created FLINK-33684:


 Summary: Improve the retry strategy in CollectResultFetcher
 Key: FLINK-33684
 URL: https://issues.apache.org/jira/browse/FLINK-33684
 Project: Flink
  Issue Type: Sub-task
  Components: API / DataStream
Reporter: xiangyu feng


Currently  CollectResultFetcher use a fixed retry interval.
{code:java}
private void sleepBeforeRetry() {
if (retryMillis <= 0) {
return;
}

try {
// TODO a more proper retry strategy?
Thread.sleep(retryMillis);
} catch (InterruptedException e) {
LOG.warn("Interrupted when sleeping before a retry", e);
}
} {code}
This can be improved with a WaitStrategy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-379: Dynamic source parallelism inference for batch jobs

[Xia Sun]

Hi everyone,

Thanks for all the comments! I will initiate the vote tomorrow if there is
no further discussion.

Best,
Xia

Leonard Xu  于2023年11月24日周五 18:50写道：

> Thanks Xia and Zhu Zhu for driving this work,
>
> It will help unify the parallelism inference for all operators of batch
> job, the updated FLIP looks good to me.
>
> Best,
> Leonard
>
>
> > 2023年11月24日 下午5:53，Xia Sun  写道：
> >
> > Hi all,
> > Offline discussed with Zhu Zhu and Leonard Xu and we have reached the
> > following three points of consensus:
> >
> > 1. Rename the interface method Context#getMaxSourceParallelism proposed
> by
> > the FLIP to Context#getParallelismInferenceUpperBound, to make the
> meaning
> > of the method clearer. See [1] for details.
> >
> > 2. We provide a more detailed explanation of the effective priority of
> the
> > dynamic source parallelism inference proposed by this FLIP and the order
> of
> > values for the upper bound of source parallelism. We also point out the
> > current support and limitations of the AdaptiveBatchScheduler regarding
> > source parallelism inference. See [2] for details.
> >
> > 3. This FLIP will only focus on the framework-level implementation and
> will
> > prioritize the implementation of FileSource as an example of the new
> > interface proposed by the FLIP. The HiveSource, due to its existing
> static
> > parallelism dynamic inference, and changes in default values for
> > configuration items such as `table.exec.hive.infer-source-parallelism`,
> > requires a more detailed migration plan, as well as more comprehensive
> > design and discussion. It is not suitable as part of this FLIP and needs
> a
> > separate FLIP. Therefore, we have removed the HiveSource part from this
> > FLIP.
> >
> > Thanks again to everyone who participated in the discussion.
> > Looking forward to your continued feedback.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-379%3A+Dynamic+source+parallelism+inference+for+batch+jobs#FLIP379:Dynamicsourceparallelisminferenceforbatchjobs-IntroduceDynamicParallelismInferenceinterfaceforSource
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-379%3A+Dynamic+source+parallelism+inference+for+batch+jobs#FLIP379:Dynamicsourceparallelisminferenceforbatchjobs-ConfigurationBehaviorChanges
> >
> > Best,
> > Xia
> >
> > Leonard Xu  于2023年11月22日周三 18:37写道：
> >
> >> Thanks Xia for the  reply, sorry for the late reply.
> >>
> >>> Thanks for pointing out the issue, the current wording does indeed seem
> >> to
> >>> be confusing. It involves the existing implementation of the
> >>> AdaptiveBatchScheduler, where the dynamically inferred parallelism
> cannot
> >>> exceed the JobVertex's maxParallelism (which is typically set to either
> >> the
> >>> global default max parallelism or the user-specified JobVertex max
> >>> parallelism), so the flip maintains the logic. I have modified the flip
> >> to
> >>> avoid confusion as much as possible.
> >>
> >> I didn’t see the change part in this FLIP, could you check it?
> >>
> >>> We can use Configuration::getOptional to check if the user has
> configured
> >>> the
> >> `execution.batch.adaptive.auto-parallelism.default-source-parallelism`.
> >>
> >> Using Readable#getOptional(ConfigOption option) makes sense to me.
> >>
> >>> As a follow-up task, we may have a dedicated discussion in the future
> to
> >>> see if we need to change the default value of
> >>> `table.exec.hive.infer-source-parallelism` to false. Before then, user
> >> can
> >>> manually set `table.exec.hive.infer-source-parallelism` to false to
> >> enable
> >>> dynamic parallelism inference, and use
> >>> `execution.batch.adaptive.auto-parallelism.default-source-parallelism`
> to
> >>> replace `table.exec.hive.infer-source-parallelism.max` as the
> parallelism
> >>> inference upper bound. I have updated both the Flip's
> >>> DynamicParallelismInference interface implementation and Migration Plan
> >>> modules to illustrate this.
> >>
> >> In my opinion, moving HiveSource to subsequent discussion is not OK, see
> >> my explanation: HiveSource supports dynamic source parallel inference is
> >> one part of the FLIP implementation, it looks like that we introduce a
> >> configuration
> >> `execution.batch.adaptive.auto-parallelism.default-source-parallelism`
> >> which conflicts with existing configuration `table.exec.hive.infer-
> >> source-parallelism`. But we do not provide conflict resolution in the
> >> current flip, just postpone the work to future discussions, this
> uncertain
> >> state is likely to cause subsequent discussions to be shelved from past
> >> community work experience. I’d suggest we make this part clear in this
> FLIP.
> >>
> >>
> >> Best,
> >> Leonard
> >>
> >>
> >>>
> >>>
> >>> Leonard Xu  于2023年11月16日周四 12:36写道：
> >>>
>  Thanks Xia for the detailed reply.
> 
> >> `How user disable the parallelism inference if they want to use
> fixed
>  source parallelism?`
> >> `Could you explain the 

Re: [VOTE] Release flink-connector-aws, v4.2.0 release candidate #1

[Leonard Xu]

Thanks Danny for driving this, sorry for late verification.

+1 (binding)

- built from source code succeeded
- verified signatures
- verified hashsums 
- checked the contents contains jar and pom files in apache repo 
- checked Github release tag 
- reviewed the web PR with minor comment

Best,
Leonard



> 2023年11月27日 上午1:13，Ahmed Hamdy  写道：
> 
> Hi Danny
> +1 (non-binding)
> 
> 
> - Verified signatures and checksums
> - verified no binaries exists in archive
> - built source
> - reviewed web PR
> - Ran E2E example with Kinesis, Firehose & DynamoDB datastream connectors.
> 
> Best Regards
> Ahmed Hamdy
> 
> 
> On Fri, 24 Nov 2023 at 08:38, Martijn Visser 
> wrote:
> 
>> Hi Danny,
>> 
>> Thanks for driving this.
>> 
>> +1 (binding)
>> 
>> - Validated hashes
>> - Verified signature
>> - Verified that no binaries exist in the source archive
>> - Build the source with Maven
>> - Verified licenses
>> - Verified web PRs
>> 
>> On Mon, Nov 20, 2023 at 12:29 PM Jiabao Sun
>>  wrote:
>>> 
>>> Thanks Danny for driving the release,
>>> 
>>> +1 (non-binding)
>>> 
>>> - built from source code succeeded
>>> - verified signatures
>>> - verified hashsums
>>> - checked release notes
>>> 
>>> Best,
>>> Jiabao
>>> 
>>> 
 2023年11月20日 19:11，Danny Cranmer  写道：
 
 Hello all,
 
 +1 (binding).
 
 - Release notes look good
 - Signatures and checksums match
 - There are no binaries in the source archive
 - pom versions are correct
 - Tag is present in Github
 - CI passes against FLink 1.17 and 1.18
 - Source build and tests pass
 
 Thanks,
 Danny
 
 On Wed, Nov 1, 2023 at 1:15 AM mystic lama 
>> wrote:
 
> +1 (non-binding)
> 
> - validated shasum
> - verified build
>  - Java 8   - build good and all test cases pass
>  - Java 11 - build good and all test cases pass
> 
> Observations: got test failures with Java 17, something to look for in
> future
> 
> On Tue, 31 Oct 2023 at 08:42, Danny Cranmer 
> wrote:
> 
>> Hi everyone,
>> 
>> Please review and vote on release candidate #1 for the version
>> 4.2.0, as
>> follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>> 
>> The complete staging area is available for your review, which
>> includes:
>> * JIRA release notes [1],
>> * the official Apache source release to be deployed to
>> dist.apache.org
>> [2],
>> which are signed with the key with fingerprint
>> 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag v4.2.0-rc1 [5],
>> * website pull request listing the new release [6].
>> * A link to the CI run on the release tag [7]
>> 
>> The vote will be open for at least 72 hours. It is adopted by
>> majority
>> approval, with at least 3 PMC affirmative votes.
>> 
>> Thanks,
>> Danny
>> 
>> [1]
>> 
>> 
> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353011
>> [2]
>> 
> 
>> https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-4.2.0-rc1
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4]
>> 
>> https://repository.apache.org/content/repositories/orgapacheflink-1665/
>> [5]
> https://github.com/apache/flink-connector-aws/releases/tag/v4.2.0-rc1
>> [6] https://github.com/apache/flink-web/pull/693
>> [7]
> https://github.com/apache/flink-connector-aws/actions/runs/6707962074
>> 
> 
>>> 
>> 

[jira] [Created] (FLINK-33683) Improve the performance of submitting jobs and fetching results to a running flink cluster

[xiangyu feng (Jira)]

xiangyu feng created FLINK-33683:


 Summary: Improve the performance of submitting jobs and fetching 
results to a running flink cluster
 Key: FLINK-33683
 URL: https://issues.apache.org/jira/browse/FLINK-33683
 Project: Flink
  Issue Type: Improvement
  Components: Client / Job Submission, Table SQL / Client
Reporter: xiangyu feng


There is now a lot of unnecessary overhead involved in submitting jobs and 
fetching results to a long-running flink cluster. This works well for streaming 
and batch job, because in these scenarios users will not frequently submit jobs 
and fetch result to a running cluster.

 

But in OLAP scenario, users will continuously submit lots of short-lived jobs 
to the running cluster. In this situation, these overhead will have a huge 
impact on the E2E performance.  Here are some examples of unnecessary overhead:
 * Each `RemoteExecutor` will create a new `StandaloneClusterDescriptor` when 
executing a job on the same remote cluster
 * `StandaloneClusterDescriptor` will always create a new `RestClusterClient` 
when retrieving an existing Flink Cluster
 * Each `RestClusterClient` will create a new `ClientHighAvailabilityServices` 
which might contains a resource-consuming ha client(ZKClient or KubeClient) and 
a time-consuming leader retrieval operation
 * `RestClient` will create a new connection for every request which costs 
extra connection establishment time

 

Therefore, I suggest creating this ticket and following subtasks to improve 
this performance. This ticket is also relates to  FLINK-25318.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33682) Reuse source operator input records/bytes metrics for SourceOperatorStreamTask

[Zhanghao Chen (Jira)]

Zhanghao Chen created FLINK-33682:
-

 Summary: Reuse source operator input records/bytes metrics for 
SourceOperatorStreamTask
 Key: FLINK-33682
 URL: https://issues.apache.org/jira/browse/FLINK-33682
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Metrics
Reporter: Zhanghao Chen


For SourceOperatorStreamTask, source opeartor is the head operator that takes 
input. We can directly reuse source operator input records/bytes metrics for it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33681) Display source/sink numRecordsIn/Out & numBytesIn/Out on UI

[Zhanghao Chen (Jira)]

Zhanghao Chen created FLINK-33681:
-

 Summary: Display source/sink numRecordsIn/Out & numBytesIn/Out on 
UI
 Key: FLINK-33681
 URL: https://issues.apache.org/jira/browse/FLINK-33681
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Metrics
Reporter: Zhanghao Chen
 Attachments: image-2023-11-29-13-26-15-176.png

Currently, the numRecordsIn & numBytesIn metrics for sources and the 
numRecordsOut & numBytesOut metrics for sinks are always 0 on the Flink web 
dashboard.

[FLINK-11576|https://issues.apache.org/jira/browse/FLINK-11576] brings us these 
metrics on the opeartor level, but it does not integrate them on the task 
level. On the other hand, the summay metrics on the job overview page is based 
on the task level I/O metrics. As a result, even though new connectors 
supporting FLIP-33 metrics will report operator-level I/O metrics, we still 
cannot see the metrics on dashboard.

This ticket serves as an umbrella issue to integrate standard source/sink I/O 
metrics with the corresponding task I/O metrics. 

!image-2023-11-29-13-26-15-176.png|width=590,height=252!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33680) Failed to build document with docke

[wangshiheng (Jira)]

wangshiheng created FLINK-33680:
---

 Summary: Failed to build document with docke
 Key: FLINK-33680
 URL: https://issues.apache.org/jira/browse/FLINK-33680
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.19.0
Reporter: wangshiheng


Follow the documentation, the documentation comes from 
[https://github.com/apache/flink/blob/master/docs/README.md]

 

The implementation results are as follows:
{code:java}
[root@bigdatadev workpace]# git clone https://github.com/section9-lab/flink.git
...

[root@bigdatadev workpace]# cd flink/docs/
[root@bigdatadev docs]# ./setup_docs.sh


[root@bigdatadev docs]# docker pull jakejarvis/hugo-extended:latest
latest: Pulling from jakejarvis/hugo-extended
Digest: sha256:f659daa3b52693d8f6fc380e4fc5d0d3faf5b9c25ef260244ff67625c59c45a7
Status: Image is up to date for jakejarvis/hugo-extended:latest
docker.io/jakejarvis/hugo-extended:latest


[root@bigdatadev docs]#  docker run -v $(pwd):/src -p 1313:1313 
jakejarvis/hugo-extended:latest server --buildDrafts --buildFuture --bind 
0.0.0.0
Start building sites … 
hugo v0.113.0-085c1b3d614e23d218ebf9daad909deaa2390c9a+extended linux/amd64 
BuildDate=2023-06-05T15:04:51Z VendorInfo=docker
Built in 515 ms
Error: error building site: assemble: "/src/content/_index.md:36:1": failed to 
extract shortcode: template for shortcode "columns" not found
 {code}
 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] Apache Flink 1.17.2 released

[Yun Tang]

The Apache Flink community is very happy to announce the release of Apache 
Flink 1.17.2, which is the second bugfix release for the Apache Flink 1.17 
series.

Apache Flink® Is a framework and distributed processing engine for stateful 
computations over unbounded and bounded data streams. Flink has been designed 
to run in all common cluster environments, perform computations at in-memory 
speed and at any scale.

The release is available for download at:
https://flink.apache.org/downloads.html

Please check out the release blog post for an overview of the improvements for 
this bugfix release:
https://flink.apache.org/2023/11/29/apache-flink-1.17.2-release-announcement/

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353260

We would like to thank all contributors of the Apache Flink community who made 
this release possible!


Feel free to reach out to the release managers (or respond to this thread) with 
feedback on the release process. Our goal is to constantly improve the release 
process. Feedback on what could be improved or things that didn't go so well 
are appreciated.


Regards,
Release Manager

Re: [DISCUSS] Towards flink-shaded release 18.0

[Sergey Nuyanzin]

Thanks a lot for the help Matthias

On Tue, Nov 28, 2023, 12:56 Matthias Pohl 
wrote:

> Ok, I will help out with the release.
>
> On Fri, Nov 24, 2023 at 6:45 PM Sergey Nuyanzin 
> wrote:
>
> > Thanks everyone for responses
> >
> > I've started preparation steps to make it happened by following steps [1]
> >
> > One of the preliminary steps is publishing GPG key dist.apache.org
> > I don't have write access there and as it is also mentioned at wiki page
> >
> > >Note: Only PMC members have write access to the release repository. If
> you
> > end up getting 403 errors ask on the mailing list for assistance.
> > Could someone please help me here or tell me which mailing list I should
> > ask for assistance?
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+flink-shaded+release#Creatingaflinkshadedrelease-Preparefortherelease
> >
> >
> >
> >
> > On Thu, Nov 23, 2023 at 11:25 AM Matthias Pohl
> >  wrote:
> >
> > > +1
> > >
> > > Thanks for driving this, Sergey. We could also think of removing the
> > > ZooKeeper versions 3.5 and 3.6, I guess. 3.6 reached EOL in 2022. Flink
> > > 1.17 already switched to 3.7 as the default version.
> > >
> > > On Thu, Nov 23, 2023 at 9:31 AM Jing Ge 
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Does it make sense to use 32.1.3-jre Guava?
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Thu, Nov 23, 2023 at 8:42 AM Yun Tang  wrote:
> > > >
> > > > > +1, thanks Sergey for driving this work.
> > > > >
> > > > > Best
> > > > > Yun Tang
> > > > > 
> > > > > From: Leonard Xu 
> > > > > Sent: Thursday, November 23, 2023 9:38
> > > > > To: dev 
> > > > > Subject: Re: [DISCUSS] Towards flink-shaded release 18.0
> > > > >
> > > > > +1, thanks Sergey for driving this.
> > > > >
> > > > > > 2023年11月23日 上午5:46，Martijn Visser  写道：
> > > > > >
> > > > > > +1. More than happy to help :)
> > > > > >
> > > > > > On Wed, Nov 22, 2023 at 9:16 PM Sergey Nuyanzin <
> > snuyan...@gmail.com
> > > >
> > > > > wrote:
> > > > > >>
> > > > > >> Hi everyone,
> > > > > >>
> > > > > >> I would like to start discussion about creating a new 18 release
> > for
> > > > > >> flink-shaded[1].
> > > > > >>
> > > > > >> Among others it brings fix for ZooKeeper CVE [2],
> > > > > >> a couple of Guava CVEs [3], [4]
> > > > > >> and support of jdk 21 from netty mentioned in one of
> > > > > >> the java 21[5] support subtasks [6]
> > > > > >>
> > > > > >> Also making a release now will allow to have enough time
> > > > > >> for testing before Flink 1.19 release.
> > > > > >>
> > > > > >> I would volunteer to make the release happen
> > > > > >> however probably I guess I will need some PMC help
> > > > > >>
> > > > > >>
> > > > > >> [1] https://github.com/apache/flink-shaded
> > > > > >> [2] https://nvd.nist.gov/vuln/detail/CVE-2023-44981
> > > > > >> [3]
> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976
> > > > > >> [4]
> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908
> > > > > >> [5] https://issues.apache.org/jira/browse/FLINK-33163
> > > > > >> [6] https://issues.apache.org/jira/browse/FLINK-1
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Best regards,
> > > > > >> Sergey
> > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Sergey
> >
>

[jira] [Created] (FLINK-33679) RestoreMode uses NO_CLAIM as default instead of LEGACY

[junzhong qin (Jira)]

junzhong qin created FLINK-33679:


 Summary: RestoreMode uses NO_CLAIM as default instead of LEGACY
 Key: FLINK-33679
 URL: https://issues.apache.org/jira/browse/FLINK-33679
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Runtime / State Backends
Reporter: junzhong qin


RestoreMode uses NO_CLAIM as default instead of LEGACY.
{code:java}
public enum RestoreMode implements DescribedEnum {
CLAIM(
"Flink will take ownership of the given snapshot. It will clean the"
+ " snapshot once it is subsumed by newer ones."),
NO_CLAIM(
"Flink will not claim ownership of the snapshot files. However it 
will make sure it"
+ " does not depend on any artefacts from the restored 
snapshot. In order to do that,"
+ " Flink will take the first checkpoint as a full one, 
which means it might"
+ " reupload/duplicate files that are part of the restored 
checkpoint."),
LEGACY(
"This is the mode in which Flink worked so far. It will not claim 
ownership of the"
+ " snapshot and will not delete the files. However, it can 
directly depend on"
+ " the existence of the files of the restored checkpoint. 
It might not be safe"
+ " to delete checkpoints that were restored in legacy mode 
");

private final String description;

RestoreMode(String description) {
this.description = description;
}

@Override
@Internal
public InlineElement getDescription() {
return text(description);
}

public static final RestoreMode DEFAULT = NO_CLAIM;
} {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Yuxin Tan]

Hi, Matthias,

Thanks for driving this.
+1 from my side.

According to the Flip, the new tests will support arm env.
I believe that's good news for arm users. I have a minor
question here. Will it be a blocker before migrating the new
tests? If not,  If not, when can we expect arm environment
support to be implemented? Thanks.

Best,
Yuxin


Márton Balassi  于2023年11月29日周三 03:09写道：

> Thanks, Matthias. Big +1 from me.
>
> On Tue, Nov 28, 2023 at 5:30 PM Matthias Pohl
>  wrote:
>
> > Thanks for the pointer. I'm planning to join that meeting.
> >
> > On Tue, Nov 28, 2023 at 4:16 PM Etienne Chauchot 
> > wrote:
> >
> > > Hi all,
> > >
> > > FYI there is the ASF infra roundtable soon. One of the subjects for
> this
> > > session is GitHub Actions. It could be worth passing by:
> > >
> > > December 6th, 2023 at 1700 UTC on the #Roundtablechannel on Slack.
> > >
> > > For information about theroundtables, and about how to join,
> > > see:https://infra.apache.org/roundtable.html
> > > 
> > >
> > > Best
> > >
> > > Etienne
> > >
> > > Le 24/11/2023 à 14:16, Maximilian Michels a écrit :
> > > > Thanks for reviving the efforts here Matthias! +1 for the transition
> > > > to GitHub Actions.
> > > >
> > > > As for ASF Infra Jenkins, it works fine. Jenkins is extremely
> > > > feature-rich. Not sure about the spare capacity though. I know that
> > > > for Apache Beam, Google donated a bunch of servers to get additional
> > > > build capacity.
> > > >
> > > > -Max
> > > >
> > > >
> > > > On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl
> > > >   wrote:
> > > >> Btw. even though we've been focusing on GitHub Actions with this
> FLIP,
> > > I'm
> > > >> curious whether somebody has experience with Apache Infra's Jenkins
> > > >> deployment. The discussion I found about Jenkins [1] is quite
> > out-dated
> > > >> (2014). I haven't worked with it myself but could imagine that there
> > are
> > > >> some features provided through plugins which are missing in GitHub
> > > Actions.
> > > >>
> > > >> [1]https://lists.apache.org/thread/vs81xdhn3q777r7x9k7wd4dyl9kvoqn4
> > > >>
> > > >> On Tue, Nov 21, 2023 at 4:19 PM Matthias Pohl<
> matthias.p...@aiven.io>
> > > >> wrote:
> > > >>
> > > >>> That's a valid point. I updated the FLIP accordingly:
> > > >>>
> > >  Currently, the secrets (e.g. for S3 access tokens) are maintained
> by
> > >  certain PMC members with access to the corresponding configuration
> > in
> > > the
> > >  Azure CI project. This responsibility will be moved to Apache
> Infra.
> > > They
> > >  are in charge of handling secrets in the Apache organization. As a
> > >  consequence, updating secrets is becoming a bit more complicated.
> > > This can
> > >  be still considered an improvement from a legal standpoint because
> > the
> > >  responsibility is transferred from an individual company (i.e.
> > > Ververica
> > >  who's the maintainer of the Azure CI project) to the Apache
> > > Foundation.
> > > >>>
> > > >>> On Tue, Nov 21, 2023 at 3:37 PM Martijn Visser<
> > > martijnvis...@apache.org>
> > > >>> wrote:
> > > >>>
> > >  Hi Matthias,
> > > 
> > >  Thanks for the write-up and for the efforts on this. I really hope
> > >  that we can move away from Azure towards GHA for a better
> > integration
> > >  as well (directly seeing if a PR can be merged due to CI passing
> for
> > >  example).
> > > 
> > >  The one thing I'm missing in the FLIP is how we would setup the
> > >  secrets for the nightly runs (for the S3 tests, potential tests
> with
> > >  external services etc). My guess is we need to provide the secret
> to
> > >  ASF Infra and then we would be able to refer to them in a
> pipeline?
> > > 
> > >  Best regards,
> > > 
> > >  Martijn
> > > 
> > >  On Tue, Nov 21, 2023 at 3:05 PM Matthias Pohl
> > >    wrote:
> > > > I realized that I mixed up FLIP IDs. FLIP-395 is already reserved
> > > [1]. I
> > > > switched to FLIP-396 [2] for the sake of consistency. 8)
> > > >
> > > > [1]
> > https://lists.apache.org/thread/wjd3nbvg6nt93lb0sd52f0lzls6559tv
> > > > [2]
> > > >
> > > 
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Migration+to+GitHub+Actions
> > > > On Tue, Nov 21, 2023 at 2:58 PM Matthias Pohl<
> > matthias.p...@aiven.io
> > > >
> > > > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> The Flink community discussed migrating from Azure CI to GitHub
> > >  Actions
> > > >> quite some time ago [1]. The efforts around that stalled due to
> > >  limitations
> > > >> around self-hosted runner support from Apache Infra’s side.
> There
> > >  were some
> > > >> recent developments on that topic. Apache Infra is experimenting
> > > with
> > > >> ephemeral runners now which might enable us to move ahead with
> > > GitHub
> > > >> Actions.
> > > 

[jira] [Created] (FLINK-33678) Remove configuration getters/setters that return/set complex Java objects

[Junrui Li (Jira)]

Junrui Li created FLINK-33678:
-

 Summary: Remove configuration getters/setters that return/set 
complex Java objects
 Key: FLINK-33678
 URL: https://issues.apache.org/jira/browse/FLINK-33678
 Project: Flink
  Issue Type: Sub-task
  Components: API / Core
Reporter: Junrui Li


FLINK-33581/FLIP-381 Deprecate configuration getters/setters that return/set 
complex Java objects. 
In Flink 2.0 we should remove these deprecated method and fields. This change 
will prevent users from configuring their jobs by passing complex Java objects, 
encouraging them to use ConfigOption instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33677) Remove flink-conf.yaml from flink dist

[Zhu Zhu (Jira)]

Zhu Zhu created FLINK-33677:
---

 Summary: Remove flink-conf.yaml from flink dist
 Key: FLINK-33677
 URL: https://issues.apache.org/jira/browse/FLINK-33677
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Configuration
Reporter: Zhu Zhu


FLINK-33297/FLIP-366 supports parsing standard YAML files for Flink 
configuration. A new configuration file config.yaml, which should be a standard 
YAML file, is introduced.
To ensure compatibility, in Flink 1.x, the old configuration parser will still 
be used if the old configuration file flink-conf.yaml exists. Only if it does 
not exist, the new configuration file will be used.
In Flink 2.0, we should remove the old configuration file from flink dist, as 
well as the old configuration parser.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Fwd: [SUMMARY] Flink 1.19 Release Sync 11/14/2023

[Lincoln Lee]

Hi devs,

This is the summary of the second release sync of Flink 1.19:

- Features & issues tracking
There're 9 weeks until the feature freeze date, so far we've had 21 flips
and the status looks good. It is encouraged to continuously updating the
1.19 wiki page[1] for contributors.
  - GitHub Actions Migration is under discussion (FLIP-396[2]), volunteers
are welcome!
  - The flink-shaded[3] version 1.18.0 will be released soon, thanks Sergey
for driving this!

- Blockers
  - [new] external connectors compilation failure cause by FLINK-25857
, now back to the ml
discussion[4]
  - [not a blocker] FLINK-31449 Remove DeclarativeSlotManager related logic
- changed to major
  - [closed] FLINK-33531 Nightly Python fails - thanks Xingbo & Dian for
fixing this
  - [closed] FLINK-18356 flink-table-planner Exit code 137 on ci pipeline -
thanks @Matthias for getting it done

- Sync meeting
The Google Meet has been updated to: https://meet.google.com/vcx-arzs-trv.
As suggested by Xintong, we encourage attendees to fill out the topics to
be discussed at the bottom of 1.19 wiki page[1] a day in advance to make it
easier for everyone to understand the background of the topic.

The next release sync will be on December 12th, 2023.

[1] https://cwiki.apache.org/confluence/display/FLINK/1.19+Release
[2] https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb
[3] https://lists.apache.org/thread/gkly9to8vnx9wrp7tvfl39s6tm4xdc9p
[4] https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57

Best regards,
Yun, Jing, Martijn and Lincoln

[VOTE] Release flink-shaded 18.0, release candidate #1

[Sergey Nuyanzin]

Hi everyone,
Please review and vote on the release candidate #1 for the version 18.0, as
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which are signed with the key with fingerprint 1596BBF0726835D8 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-18.0-rc1" [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Sergey

[1] https://issues.apache.org/jira/projects/FLINK/versions/12353081
[2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-18.0-rc1
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1676/
[5] https://github.com/apache/flink-shaded/releases/tag/release-18.0-rc1
[6] https://github.com/apache/flink-web/pull/701

[jira] [Created] (FLINK-33676) Implement restore tests for WindowAggregate node

[Jim Hughes (Jira)]

Jim Hughes created FLINK-33676:
--

 Summary: Implement restore tests for WindowAggregate node
 Key: FLINK-33676
 URL: https://issues.apache.org/jira/browse/FLINK-33676
 Project: Flink
  Issue Type: Sub-task
Reporter: Jim Hughes
Assignee: Jim Hughes






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Márton Balassi]

Thanks, Matthias. Big +1 from me.

On Tue, Nov 28, 2023 at 5:30 PM Matthias Pohl
 wrote:

> Thanks for the pointer. I'm planning to join that meeting.
>
> On Tue, Nov 28, 2023 at 4:16 PM Etienne Chauchot 
> wrote:
>
> > Hi all,
> >
> > FYI there is the ASF infra roundtable soon. One of the subjects for this
> > session is GitHub Actions. It could be worth passing by:
> >
> > December 6th, 2023 at 1700 UTC on the #Roundtablechannel on Slack.
> >
> > For information about theroundtables, and about how to join,
> > see:https://infra.apache.org/roundtable.html
> > 
> >
> > Best
> >
> > Etienne
> >
> > Le 24/11/2023 à 14:16, Maximilian Michels a écrit :
> > > Thanks for reviving the efforts here Matthias! +1 for the transition
> > > to GitHub Actions.
> > >
> > > As for ASF Infra Jenkins, it works fine. Jenkins is extremely
> > > feature-rich. Not sure about the spare capacity though. I know that
> > > for Apache Beam, Google donated a bunch of servers to get additional
> > > build capacity.
> > >
> > > -Max
> > >
> > >
> > > On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl
> > >   wrote:
> > >> Btw. even though we've been focusing on GitHub Actions with this FLIP,
> > I'm
> > >> curious whether somebody has experience with Apache Infra's Jenkins
> > >> deployment. The discussion I found about Jenkins [1] is quite
> out-dated
> > >> (2014). I haven't worked with it myself but could imagine that there
> are
> > >> some features provided through plugins which are missing in GitHub
> > Actions.
> > >>
> > >> [1]https://lists.apache.org/thread/vs81xdhn3q777r7x9k7wd4dyl9kvoqn4
> > >>
> > >> On Tue, Nov 21, 2023 at 4:19 PM Matthias Pohl
> > >> wrote:
> > >>
> > >>> That's a valid point. I updated the FLIP accordingly:
> > >>>
> >  Currently, the secrets (e.g. for S3 access tokens) are maintained by
> >  certain PMC members with access to the corresponding configuration
> in
> > the
> >  Azure CI project. This responsibility will be moved to Apache Infra.
> > They
> >  are in charge of handling secrets in the Apache organization. As a
> >  consequence, updating secrets is becoming a bit more complicated.
> > This can
> >  be still considered an improvement from a legal standpoint because
> the
> >  responsibility is transferred from an individual company (i.e.
> > Ververica
> >  who's the maintainer of the Azure CI project) to the Apache
> > Foundation.
> > >>>
> > >>> On Tue, Nov 21, 2023 at 3:37 PM Martijn Visser<
> > martijnvis...@apache.org>
> > >>> wrote:
> > >>>
> >  Hi Matthias,
> > 
> >  Thanks for the write-up and for the efforts on this. I really hope
> >  that we can move away from Azure towards GHA for a better
> integration
> >  as well (directly seeing if a PR can be merged due to CI passing for
> >  example).
> > 
> >  The one thing I'm missing in the FLIP is how we would setup the
> >  secrets for the nightly runs (for the S3 tests, potential tests with
> >  external services etc). My guess is we need to provide the secret to
> >  ASF Infra and then we would be able to refer to them in a pipeline?
> > 
> >  Best regards,
> > 
> >  Martijn
> > 
> >  On Tue, Nov 21, 2023 at 3:05 PM Matthias Pohl
> >    wrote:
> > > I realized that I mixed up FLIP IDs. FLIP-395 is already reserved
> > [1]. I
> > > switched to FLIP-396 [2] for the sake of consistency. 8)
> > >
> > > [1]
> https://lists.apache.org/thread/wjd3nbvg6nt93lb0sd52f0lzls6559tv
> > > [2]
> > >
> > 
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Migration+to+GitHub+Actions
> > > On Tue, Nov 21, 2023 at 2:58 PM Matthias Pohl<
> matthias.p...@aiven.io
> > >
> > > wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> The Flink community discussed migrating from Azure CI to GitHub
> >  Actions
> > >> quite some time ago [1]. The efforts around that stalled due to
> >  limitations
> > >> around self-hosted runner support from Apache Infra’s side. There
> >  were some
> > >> recent developments on that topic. Apache Infra is experimenting
> > with
> > >> ephemeral runners now which might enable us to move ahead with
> > GitHub
> > >> Actions.
> > >>
> > >> The goal is to join the trial phase for ephemeral runners and
> >  experiment
> > >> with our CI workflows in terms of stability and performance. At
> the
> >  end we
> > >> can decide whether we want to abandon Azure CI and move to GitHub
> >  Actions
> > >> or stick to the former one.
> > >>
> > >> Nico Weidner and Chesnay laid the groundwork on this topic in the
> >  past. I
> > >> picked up the work they did and continued experimenting with it in
> > my
> >  own
> > >> fork XComp/flink [2] the past few weeks. The workflows are in a
> > state
> >  where
> > >> I think that we start moving the 

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Matthias Pohl]

Thanks for the pointer. I'm planning to join that meeting.

On Tue, Nov 28, 2023 at 4:16 PM Etienne Chauchot 
wrote:

> Hi all,
>
> FYI there is the ASF infra roundtable soon. One of the subjects for this
> session is GitHub Actions. It could be worth passing by:
>
> December 6th, 2023 at 1700 UTC on the #Roundtablechannel on Slack.
>
> For information about theroundtables, and about how to join,
> see:https://infra.apache.org/roundtable.html
> 
>
> Best
>
> Etienne
>
> Le 24/11/2023 à 14:16, Maximilian Michels a écrit :
> > Thanks for reviving the efforts here Matthias! +1 for the transition
> > to GitHub Actions.
> >
> > As for ASF Infra Jenkins, it works fine. Jenkins is extremely
> > feature-rich. Not sure about the spare capacity though. I know that
> > for Apache Beam, Google donated a bunch of servers to get additional
> > build capacity.
> >
> > -Max
> >
> >
> > On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl
> >   wrote:
> >> Btw. even though we've been focusing on GitHub Actions with this FLIP,
> I'm
> >> curious whether somebody has experience with Apache Infra's Jenkins
> >> deployment. The discussion I found about Jenkins [1] is quite out-dated
> >> (2014). I haven't worked with it myself but could imagine that there are
> >> some features provided through plugins which are missing in GitHub
> Actions.
> >>
> >> [1]https://lists.apache.org/thread/vs81xdhn3q777r7x9k7wd4dyl9kvoqn4
> >>
> >> On Tue, Nov 21, 2023 at 4:19 PM Matthias Pohl
> >> wrote:
> >>
> >>> That's a valid point. I updated the FLIP accordingly:
> >>>
>  Currently, the secrets (e.g. for S3 access tokens) are maintained by
>  certain PMC members with access to the corresponding configuration in
> the
>  Azure CI project. This responsibility will be moved to Apache Infra.
> They
>  are in charge of handling secrets in the Apache organization. As a
>  consequence, updating secrets is becoming a bit more complicated.
> This can
>  be still considered an improvement from a legal standpoint because the
>  responsibility is transferred from an individual company (i.e.
> Ververica
>  who's the maintainer of the Azure CI project) to the Apache
> Foundation.
> >>>
> >>> On Tue, Nov 21, 2023 at 3:37 PM Martijn Visser<
> martijnvis...@apache.org>
> >>> wrote:
> >>>
>  Hi Matthias,
> 
>  Thanks for the write-up and for the efforts on this. I really hope
>  that we can move away from Azure towards GHA for a better integration
>  as well (directly seeing if a PR can be merged due to CI passing for
>  example).
> 
>  The one thing I'm missing in the FLIP is how we would setup the
>  secrets for the nightly runs (for the S3 tests, potential tests with
>  external services etc). My guess is we need to provide the secret to
>  ASF Infra and then we would be able to refer to them in a pipeline?
> 
>  Best regards,
> 
>  Martijn
> 
>  On Tue, Nov 21, 2023 at 3:05 PM Matthias Pohl
>    wrote:
> > I realized that I mixed up FLIP IDs. FLIP-395 is already reserved
> [1]. I
> > switched to FLIP-396 [2] for the sake of consistency. 8)
> >
> > [1]https://lists.apache.org/thread/wjd3nbvg6nt93lb0sd52f0lzls6559tv
> > [2]
> >
> 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Migration+to+GitHub+Actions
> > On Tue, Nov 21, 2023 at 2:58 PM Matthias Pohl >
> > wrote:
> >
> >> Hi everyone,
> >>
> >> The Flink community discussed migrating from Azure CI to GitHub
>  Actions
> >> quite some time ago [1]. The efforts around that stalled due to
>  limitations
> >> around self-hosted runner support from Apache Infra’s side. There
>  were some
> >> recent developments on that topic. Apache Infra is experimenting
> with
> >> ephemeral runners now which might enable us to move ahead with
> GitHub
> >> Actions.
> >>
> >> The goal is to join the trial phase for ephemeral runners and
>  experiment
> >> with our CI workflows in terms of stability and performance. At the
>  end we
> >> can decide whether we want to abandon Azure CI and move to GitHub
>  Actions
> >> or stick to the former one.
> >>
> >> Nico Weidner and Chesnay laid the groundwork on this topic in the
>  past. I
> >> picked up the work they did and continued experimenting with it in
> my
>  own
> >> fork XComp/flink [2] the past few weeks. The workflows are in a
> state
>  where
> >> I think that we start moving the relevant code into Flink’s
>  repository.
> >> Example runs for the basic workflow [3] and the extended (nightly)
>  workflow
> >> [4] are provided.
> >>
> >> This will bring a few more changes to the Flink contributors. That
> is
>  why
> >> I wanted to bring this discussion to the mailing list first. I did a
>  write
> >> up on (hopefully) all related 

[jira] [Created] (FLINK-33675) Serialize ValueLiteralExpressions into SQL

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-33675:


 Summary: Serialize ValueLiteralExpressions into SQL
 Key: FLINK-33675
 URL: https://issues.apache.org/jira/browse/FLINK-33675
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.19.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Etienne Chauchot]

Hi all,

FYI there is the ASF infra roundtable soon. One of the subjects for this 
session is GitHub Actions. It could be worth passing by:


December 6th, 2023 at 1700 UTC on the #Roundtablechannel on Slack.

For information about theroundtables, and about how to join, 
see:https://infra.apache.org/roundtable.html 



Best

Etienne

Le 24/11/2023 à 14:16, Maximilian Michels a écrit :

Thanks for reviving the efforts here Matthias! +1 for the transition
to GitHub Actions.

As for ASF Infra Jenkins, it works fine. Jenkins is extremely
feature-rich. Not sure about the spare capacity though. I know that
for Apache Beam, Google donated a bunch of servers to get additional
build capacity.

-Max


On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl
  wrote:

Btw. even though we've been focusing on GitHub Actions with this FLIP, I'm
curious whether somebody has experience with Apache Infra's Jenkins
deployment. The discussion I found about Jenkins [1] is quite out-dated
(2014). I haven't worked with it myself but could imagine that there are
some features provided through plugins which are missing in GitHub Actions.

[1]https://lists.apache.org/thread/vs81xdhn3q777r7x9k7wd4dyl9kvoqn4

On Tue, Nov 21, 2023 at 4:19 PM Matthias Pohl
wrote:


That's a valid point. I updated the FLIP accordingly:


Currently, the secrets (e.g. for S3 access tokens) are maintained by
certain PMC members with access to the corresponding configuration in the
Azure CI project. This responsibility will be moved to Apache Infra. They
are in charge of handling secrets in the Apache organization. As a
consequence, updating secrets is becoming a bit more complicated. This can
be still considered an improvement from a legal standpoint because the
responsibility is transferred from an individual company (i.e. Ververica
who's the maintainer of the Azure CI project) to the Apache Foundation.


On Tue, Nov 21, 2023 at 3:37 PM Martijn Visser
wrote:


Hi Matthias,

Thanks for the write-up and for the efforts on this. I really hope
that we can move away from Azure towards GHA for a better integration
as well (directly seeing if a PR can be merged due to CI passing for
example).

The one thing I'm missing in the FLIP is how we would setup the
secrets for the nightly runs (for the S3 tests, potential tests with
external services etc). My guess is we need to provide the secret to
ASF Infra and then we would be able to refer to them in a pipeline?

Best regards,

Martijn

On Tue, Nov 21, 2023 at 3:05 PM Matthias Pohl
  wrote:

I realized that I mixed up FLIP IDs. FLIP-395 is already reserved [1]. I
switched to FLIP-396 [2] for the sake of consistency. 8)

[1]https://lists.apache.org/thread/wjd3nbvg6nt93lb0sd52f0lzls6559tv
[2]


https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Migration+to+GitHub+Actions

On Tue, Nov 21, 2023 at 2:58 PM Matthias Pohl
wrote:


Hi everyone,

The Flink community discussed migrating from Azure CI to GitHub

Actions

quite some time ago [1]. The efforts around that stalled due to

limitations

around self-hosted runner support from Apache Infra’s side. There

were some

recent developments on that topic. Apache Infra is experimenting with
ephemeral runners now which might enable us to move ahead with GitHub
Actions.

The goal is to join the trial phase for ephemeral runners and

experiment

with our CI workflows in terms of stability and performance. At the

end we

can decide whether we want to abandon Azure CI and move to GitHub

Actions

or stick to the former one.

Nico Weidner and Chesnay laid the groundwork on this topic in the

past. I

picked up the work they did and continued experimenting with it in my

own

fork XComp/flink [2] the past few weeks. The workflows are in a state

where

I think that we start moving the relevant code into Flink’s

repository.

Example runs for the basic workflow [3] and the extended (nightly)

workflow

[4] are provided.

This will bring a few more changes to the Flink contributors. That is

why

I wanted to bring this discussion to the mailing list first. I did a

write

up on (hopefully) all related topics in FLIP-395 [5].

I’m looking forward to your feedback.

Matthias

[1]https://lists.apache.org/thread/vcyx2nx0mhklqwm827vgykv8pc54gg3k

[2]https://github.com/XComp/flink/actions

[3]https://github.com/XComp/flink/actions/runs/6926309782

[4]https://github.com/XComp/flink/actions/runs/6927443941

[5]


https://cwiki.apache.org/confluence/display/FLINK/FLIP-395%3A+Migration+to+GitHub+Actions


--

[image: Aiven]

*Matthias Pohl*
Opensource Software Engineer, *Aiven*
matthias.p...@aiven.io  |  +49 170 9869525
aiven.io|

<

https://twitter.com/aiven_io>

*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSSION] flink-connector-shared-utils release process

[Etienne Chauchot]

Hi all,

I just updated the doc: 
https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development


Best

Etienne

Le 27/11/2023 à 12:43, Etienne Chauchot a écrit :


Sure!

Le 23/11/2023 à 02:57, Leonard Xu a écrit :

Thanks Etienne for driving this.


- a flink-connector-shared-utils-*test* clone repo and a 
*io.github.user.flink*:flink-connector-parent custom artifact to be able to 
directly commit and install the artifact in the CI
- a custom ci script that does the cloning and mvn install in the ci.yml github 
action script for testing with the new flink-connector-parent artifact
If people agree on the process and location

+1 for the process and location, could we also add a short paragraph and the 
location link as well in [1] to remind connector developers?

Best,
Leonard

[1]https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development

Re: [VOTE] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

[Jing Ge]

+1(binding)
Thanks!

Best regards,
Jing

On Tue, Nov 28, 2023 at 1:00 PM weijie guo 
wrote:

> Thanks Junrui for driving this!
>
> +1(binding)
>
> Best regards,
>
> Weijie
>
>
> Zhanghao Chen  于2023年11月28日周二 17:29写道：
>
> > +1 (non-binding)
> >
> > Best,
> > Zhanghao Chen
> > 
> > From: Junrui Lee 
> > Sent: Tuesday, November 28, 2023 12:34
> > To: dev@flink.apache.org 
> > Subject: [VOTE] FLIP-391: Deprecate RuntimeContext#getExecutionConfig
> >
> > Hi everyone,
> >
> > Thank you to everyone for the feedback on FLIP-391: Deprecate
> > RuntimeContext#getExecutionConfig[1] which has been discussed in this
> > thread [2].
> >
> > I would like to start a vote for it. The vote will be open for at least
> 72
> > hours unless there is an objection or not enough votes.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278465937
> > [2]https://lists.apache.org/thread/cv4x3372g5txw1j20r5l1vwlqrvjoqq5
> >
>

[DISCUSS] FLIP-392: Deprecate the Legacy Group Window Aggregation

[Xuyang]

Hi all.
I'd like to start a discussion of FLIP-392: Deprecate the Legacy Group Window 
Aggregation.


Although the current Flink SQL Window Aggregation documentation[1] indicates 
that the legacy Group Window Aggregation
syntax has been deprecated, the new Window TVF Aggregation syntax has not fully 
covered all of the features of the legacy one.


Compared to Group Window Aggergation, Window TVF Aggergation has several 
advantages, such as two-stage optimization, 
support for standard GROUPING SET syntax, and so on. However, it needs to 
supplement and enrich the following features.


1. Support for SESSION Window TVF Aggregation
2. Support for consuming CDC stream
3. Support for HOP window size with non-integer step length
4. Support for configurations such as early fire, late fire and allow lateness
(which are internal experimental configurations in Group Window Aggregation and 
not public to users yet.)
5. Unification of the Window TVF Aggregation operator in runtime at the 
implementation layer
(In the long term, the cost to maintain the operators about Window TVF 
Aggregation and Group Window Aggregation is too expensive.)


This flip aims to continue the unfinished work in FLIP-145[2], which is to 
fully enable the capabilities of Window TVF Aggregation
 and officially deprecate the legacy syntax Group Window Aggregation, to 
prepare for the removal of the legacy one in Flink 2.0. 


I have already done some preliminary POC to validate the feasibility of the 
related work in this flip as follows.
1. POC for SESSION Window TVF Aggregation [3]
2. POC for CUMULATE in Group Window Aggregation operator [4]
3. POC for consuming CDC stream in Window Aggregation operator [5]


Looking forward to your feedback and thoughts!



[1] 
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/window-agg/

[2] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-145%3A+Support+SQL+windowing+table-valued+function#FLIP145:SupportSQLwindowingtablevaluedfunction-SessionWindows
[3] https://github.com/xuyangzhong/flink/tree/FLINK-24024
[4] 
https://github.com/xuyangzhong/flink/tree/poc_legacy_group_window_agg_cumulate
[5] https://github.com/xuyangzhong/flink/tree/poc_window_agg_consumes_cdc_stream



--

Best！
Xuyang

Re: [VOTE] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

[weijie guo]

Thanks Junrui for driving this!

+1(binding)

Best regards,

Weijie


Zhanghao Chen  于2023年11月28日周二 17:29写道：

> +1 (non-binding)
>
> Best,
> Zhanghao Chen
> 
> From: Junrui Lee 
> Sent: Tuesday, November 28, 2023 12:34
> To: dev@flink.apache.org 
> Subject: [VOTE] FLIP-391: Deprecate RuntimeContext#getExecutionConfig
>
> Hi everyone,
>
> Thank you to everyone for the feedback on FLIP-391: Deprecate
> RuntimeContext#getExecutionConfig[1] which has been discussed in this
> thread [2].
>
> I would like to start a vote for it. The vote will be open for at least 72
> hours unless there is an objection or not enough votes.
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278465937
> [2]https://lists.apache.org/thread/cv4x3372g5txw1j20r5l1vwlqrvjoqq5
>

Re: [DISCUSS] Towards flink-shaded release 18.0

[Matthias Pohl]

Ok, I will help out with the release.

On Fri, Nov 24, 2023 at 6:45 PM Sergey Nuyanzin  wrote:

> Thanks everyone for responses
>
> I've started preparation steps to make it happened by following steps [1]
>
> One of the preliminary steps is publishing GPG key dist.apache.org
> I don't have write access there and as it is also mentioned at wiki page
>
> >Note: Only PMC members have write access to the release repository. If you
> end up getting 403 errors ask on the mailing list for assistance.
> Could someone please help me here or tell me which mailing list I should
> ask for assistance?
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+flink-shaded+release#Creatingaflinkshadedrelease-Preparefortherelease
>
>
>
>
> On Thu, Nov 23, 2023 at 11:25 AM Matthias Pohl
>  wrote:
>
> > +1
> >
> > Thanks for driving this, Sergey. We could also think of removing the
> > ZooKeeper versions 3.5 and 3.6, I guess. 3.6 reached EOL in 2022. Flink
> > 1.17 already switched to 3.7 as the default version.
> >
> > On Thu, Nov 23, 2023 at 9:31 AM Jing Ge 
> > wrote:
> >
> > > +1
> > >
> > > Does it make sense to use 32.1.3-jre Guava?
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Thu, Nov 23, 2023 at 8:42 AM Yun Tang  wrote:
> > >
> > > > +1, thanks Sergey for driving this work.
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Leonard Xu 
> > > > Sent: Thursday, November 23, 2023 9:38
> > > > To: dev 
> > > > Subject: Re: [DISCUSS] Towards flink-shaded release 18.0
> > > >
> > > > +1, thanks Sergey for driving this.
> > > >
> > > > > 2023年11月23日 上午5:46，Martijn Visser  写道：
> > > > >
> > > > > +1. More than happy to help :)
> > > > >
> > > > > On Wed, Nov 22, 2023 at 9:16 PM Sergey Nuyanzin <
> snuyan...@gmail.com
> > >
> > > > wrote:
> > > > >>
> > > > >> Hi everyone,
> > > > >>
> > > > >> I would like to start discussion about creating a new 18 release
> for
> > > > >> flink-shaded[1].
> > > > >>
> > > > >> Among others it brings fix for ZooKeeper CVE [2],
> > > > >> a couple of Guava CVEs [3], [4]
> > > > >> and support of jdk 21 from netty mentioned in one of
> > > > >> the java 21[5] support subtasks [6]
> > > > >>
> > > > >> Also making a release now will allow to have enough time
> > > > >> for testing before Flink 1.19 release.
> > > > >>
> > > > >> I would volunteer to make the release happen
> > > > >> however probably I guess I will need some PMC help
> > > > >>
> > > > >>
> > > > >> [1] https://github.com/apache/flink-shaded
> > > > >> [2] https://nvd.nist.gov/vuln/detail/CVE-2023-44981
> > > > >> [3] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976
> > > > >> [4] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908
> > > > >> [5] https://issues.apache.org/jira/browse/FLINK-33163
> > > > >> [6] https://issues.apache.org/jira/browse/FLINK-1
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Best regards,
> > > > >> Sergey
> > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Sergey
>

[jira] [Created] (FLINK-33674) In the wmassigners module, words are misspelled

[wangshiheng (Jira)]

wangshiheng created FLINK-33674:
---

 Summary: In the wmassigners module, words are misspelled
 Key: FLINK-33674
 URL: https://issues.apache.org/jira/browse/FLINK-33674
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Runtime
Affects Versions: 1.19.0
Reporter: wangshiheng
 Fix For: 1.19.0


RowTimeMiniBatch{color:#FF}Assginer{color}Operator changed to 
RowTimeMiniBatch{color:#ff8b00}Assigner{color}Operator

RowTimeMiniBatch{color:#FF}Assginer{color}OperatorTest changed to 
RowTimeMiniBatch{color:#ff8b00}Assigner{color}OperatorTest



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33673) SizeLimits not being set on emptyDir

[Tony Garrard (Jira)]

Tony Garrard created FLINK-33673:


 Summary: SizeLimits not being set on emptyDir
 Key: FLINK-33673
 URL: https://issues.apache.org/jira/browse/FLINK-33673
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.7.0
Reporter: Tony Garrard
 Fix For: kubernetes-operator-1.8.0


The operator should set a sizeLimit on any emptyDir's it creates. See 
[https://main.kyverno.io/policies/other/a/add-emptydir-sizelimit/add-emptydir-sizelimit/.
 
|https://main.kyverno.io/policies/other/a/add-emptydir-sizelimit/add-emptydir-sizelimit/]

This issue is to set a sizeLimit. The default one in question is for artifacts. 
My initial guess at a setting would be around 512Mb



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

[Zhanghao Chen]

+1 (non-binding)

Best,
Zhanghao Chen

From: Junrui Lee 
Sent: Tuesday, November 28, 2023 12:34
To: dev@flink.apache.org 
Subject: [VOTE] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Hi everyone,

Thank you to everyone for the feedback on FLIP-391: Deprecate
RuntimeContext#getExecutionConfig[1] which has been discussed in this
thread [2].

I would like to start a vote for it. The vote will be open for at least 72
hours unless there is an objection or not enough votes.

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278465937
[2]https://lists.apache.org/thread/cv4x3372g5txw1j20r5l1vwlqrvjoqq5

[jira] [Created] (FLINK-33672) Use MapState.entries() instead of keys() and get() in over window

[Zakelly Lan (Jira)]

Zakelly Lan created FLINK-33672:
---

 Summary: Use MapState.entries() instead of keys() and get() in 
over window
 Key: FLINK-33672
 URL: https://issues.apache.org/jira/browse/FLINK-33672
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Runtime
Reporter: Zakelly Lan


In code logic related with over windows, such as 
org.apache.flink.table.runtime.operators.over.ProcTimeRangeBoundedPrecedingFunction

{code:java}
public void onTimer(
long timestamp,
KeyedProcessFunction.OnTimerContext ctx,
Collector out)
throws Exception {
//...
Iterator iter = inputState.keys().iterator();
//...
while (iter.hasNext()) {
Long elementKey = iter.next();
if (elementKey < limit) {
// element key outside of window. Retract values
List elementsRemove = inputState.get(elementKey);
// ...
}
}
//...
} {code}

As we can see, there is a combination of key iteration and get the value for 
iterated key from inputState. However for RocksDB, the key iteration calls 
entry iteration, which means actually we could replace it by entry iteration 
without introducing any extra overhead. And as a result, we could save a 
function call of get() by using getValue() of iterated entry at very low cost.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33671) SurefireBooterForkException caused test failure

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-33671:
-

 Summary: SurefireBooterForkException caused test failure
 Key: FLINK-33671
 URL: https://issues.apache.org/jira/browse/FLINK-33671
 Project: Flink
  Issue Type: Bug
  Components: Build System / CI
Affects Versions: 1.19.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54975=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11785

{code}
Nov 28 05:56:36 05:56:36.545 [ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test 
(integration-tests) on project flink-table-planner_2.12: There are test 
failures.
Nov 28 05:56:36 05:56:36.545 [ERROR] 
Nov 28 05:56:36 05:56:36.545 [ERROR] Please refer to 
/__w/1/s/flink-table/flink-table-planner/target/surefire-reports for the 
individual test results.
Nov 28 05:56:36 05:56:36.545 [ERROR] Please refer to dump files (if any exist) 
[date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Nov 28 05:56:36 05:56:36.545 [ERROR] ExecutionException Error occurred in 
starting fork, check output in log
Nov 28 05:56:36 05:56:36.545 [ERROR] 
org.apache.maven.surefire.booter.SurefireBooterForkException: 
ExecutionException Error occurred in starting fork, check output in log
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:370)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:351)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:215)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:171)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:163)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:294)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
Nov 28 05:56:36 05:56:36.546 [ERROR]at 
org.apache.maven.cli.MavenCli.execute(MavenCli.java:960)
Nov 28 05:56:36 05:56:36.547 [ERROR]at 
org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293)
Nov 28 05:56:36 05:56:36.547 [ERROR]at 
org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
Nov 28 05:56:36 05:56:36.547 [ERROR]at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Nov 28 05:56:36 05:56:36.547 [ERROR]at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Nov 28 05:56:36 05:56:36.547 [ERROR]at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Nov 28 05:56:36 05:56:36.547 [ERROR]at 
java.lang.reflect.Method.invoke(Method.java:498)
Nov 28 05:56:36 05:56:36.547 [ERROR]at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
Nov 28 05:56:36 05:56:36.547 [ERROR]at