Re: Flink 1.6.4 Issue on RocksDB incremental checkpoints and fs.default-scheme
Hi Andrea The error happens when Flink try to verify whether your local backup directory existed[1]. If you could reproduce this, would you please share your configuration to RocksDBStateBackend, and what `fs.default-scheme` have you configured. Taskmanager log with more details is also very welcome. [1] https://github.com/apache/flink/blob/6f4148180ba372a2c12c1d54bea8579350af6c98/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L2568 Best Yun Tang From: Andrea Spina Sent: Monday, July 1, 2019 20:06 To: dev@flink.apache.org Subject: Fwd: Flink 1.6.4 Issue on RocksDB incremental checkpoints and fs.default-scheme Dear community, I am running through the following issue. whenever I use rocksdb as state backend along with incremental checkpoints, I get the following error: *Caused by: java.lang.Exception: Could not materialize checkpoint 1 for operator Service Join SuperService (6/8).at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942) ... 6 moreCaused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateExceptionat java.util.concurrent.FutureTask.report(FutureTask.java:122)at java.util.concurrent.FutureTask.get(FutureTask.java:192)at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53) at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:47) at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853) ... 5 moreCaused by: java.lang.IllegalStateExceptionat org.apache.flink.util.Preconditions.checkState(Preconditions.java:179) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.runSnapshot(RocksDBKeyedStateBackend.java:2568) at java.util.concurrent.FutureTask.run(FutureTask.java:266)at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50) ... 7 more* In my case, I am able to use incremental checkopints with rocksdb as long as I disable *fs.default-scheme* property; in any other case, I get the above error. Is this a known issue? Hope this can help, -- *Andrea Spina* Head of R&D @ Radicalbit Srl Via Giovanni Battista Pirelli 11, 20124, Milano - IT
Re: Flink 1.6.4 Issue on RocksDB incremental checkpoints and fs.default-scheme
Hi Andrea Unfortunately, the tm log provided is not the task manager in which RocksDBStateBackend first failed. All tasks on this task manager are actually canceled by job manager, you could find a lot of "Attempting to cancel task" before any task failed. >From your latest description, this problem happened without any relationship >to fs.default-schema. And actually I wonder the previous error "Could not >materialize checkpoint 1 for operator Service Join SuperService (6/8)" was >whether the root cause of your job's first failover, it might be caused by >other task failure and then cancelled via JM leading to that directory cleaned >up. I think you could search your job manager's log to find the first failed task exception and locate which task manager that task run. That task manager would contain useful messages. If possible, please provide your job manager's log. Best Yun Tang From: Andrea Spina Sent: Monday, July 1, 2019 23:14 To: dev@flink.apache.org Subject: Re: Flink 1.6.4 Issue on RocksDB incremental checkpoints and fs.default-scheme Hi Yun, rocksDB configuration is set as follows: ``` RocksDB write-buffer size: 512MB RocksDB BlockSize (cache) [B/K/M]: 128MB Checkpoints directory: hdfs:///address-to-hdfs-chkp-dir:8020/flink/checkpoints enable Checkpoints: true Rocksdb cache index and filters true RocksDB thread No.: 4 Checkpoints interval: 6 RocksDB BlockSize [B/K/M]: 16KB RocksDB write-buffer count: 5 Use incremental checkpoints: true Rocksdb optimize hits: true RocksDB write-buffer number to merge: 2 ``` I use RocksDBStateBackend class, but I recorded the same result by using configuration parameter state.backend.incremental: true. Il giorno lun 1 lug 2019 alle ore 14:41 Yun Tang mailto:myas...@live.com>> ha scritto: Hi Andrea The error happens when Flink try to verify whether your local backup directory existed[1]. If you could reproduce this, would you please share your configuration to RocksDBStateBackend, and what `fs.default-scheme` have you configured. Taskmanager log with more details is also very welcome. [1] https://github.com/apache/flink/blob/6f4148180ba372a2c12c1d54bea8579350af6c98/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L2568 Best Yun Tang From: Andrea Spina mailto:andrea.sp...@radicalbit.io>> Sent: Monday, July 1, 2019 20:06 To: dev@flink.apache.org<mailto:dev@flink.apache.org> Subject: Fwd: Flink 1.6.4 Issue on RocksDB incremental checkpoints and fs.default-scheme Dear community, I am running through the following issue. whenever I use rocksdb as state backend along with incremental checkpoints, I get the following error: *Caused by: java.lang.Exception: Could not materialize checkpoint 1 for operator Service Join SuperService (6/8).at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942) ... 6 moreCaused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateExceptionat java.util.concurrent.FutureTask.report(FutureTask.java:122)at java.util.concurrent.FutureTask.get(FutureTask.java:192)at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53) at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:47) at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853) ... 5 moreCaused by: java.lang.IllegalStateExceptionat org.apache.flink.util.Preconditions.checkState(Preconditions.java:179) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.runSnapshot(RocksDBKeyedStateBackend.java:2568) at java.util.concurrent.FutureTask.run(FutureTask.java:266)at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50) ... 7 more* In my case, I am able to use incremental checkopints with rocksdb as long as I disable *fs.default-scheme* property; in any other case, I get the above error. Is this a known issue? Hope this can help, -- *Andrea Spina* Head of R&D @ Radicalbit Srl Via Giovanni Battista Pirelli 11, 20124, Milano - IT -- Andrea Spina Head of R&D @ Radicalbit Srl Via Giovanni Battista Pirelli 11, 20124, Milano - IT
Re: Flink 1.6.4 Issue on RocksDB incremental checkpoints and fs.default-scheme
Hi Andrea This should be a bug already fixed by https://issues.apache.org/jira/browse/FLINK-12042 , you could upgrade to at least 1.7.3 version to avoid this bug. Best Yun Tang From: Andrea Spina Sent: Wednesday, July 3, 2019 15:46 To: dev@flink.apache.org Subject: Re: Flink 1.6.4 Issue on RocksDB incremental checkpoints and fs.default-scheme Hi, I attached also the JM log. Thereby you can appreciate the exception. I hope that can help. As I said previously, disabling fs.default-scheme property solved my issue. cheers, Il giorno lun 1 lug 2019 alle ore 18:17 Yun Tang mailto:myas...@live.com>> ha scritto: Hi Andrea Unfortunately, the tm log provided is not the task manager in which RocksDBStateBackend first failed. All tasks on this task manager are actually canceled by job manager, you could find a lot of "Attempting to cancel task" before any task failed. >From your latest description, this problem happened without any relationship >to fs.default-schema. And actually I wonder the previous error "Could not >materialize checkpoint 1 for operator Service Join SuperService (6/8)" was >whether the root cause of your job's first failover, it might be caused by >other task failure and then cancelled via JM leading to that directory cleaned >up. I think you could search your job manager's log to find the first failed task exception and locate which task manager that task run. That task manager would contain useful messages. If possible, please provide your job manager's log. Best Yun Tang From: Andrea Spina mailto:andrea.sp...@radicalbit.io>> Sent: Monday, July 1, 2019 23:14 To: dev@flink.apache.org<mailto:dev@flink.apache.org> Subject: Re: Flink 1.6.4 Issue on RocksDB incremental checkpoints and fs.default-scheme Hi Yun, rocksDB configuration is set as follows: ``` RocksDB write-buffer size: 512MB RocksDB BlockSize (cache) [B/K/M]: 128MB Checkpoints directory: hdfs:///address-to-hdfs-chkp-dir:8020/flink/checkpoints enable Checkpoints: true Rocksdb cache index and filters true RocksDB thread No.: 4 Checkpoints interval: 6 RocksDB BlockSize [B/K/M]: 16KB RocksDB write-buffer count: 5 Use incremental checkpoints: true Rocksdb optimize hits: true RocksDB write-buffer number to merge: 2 ``` I use RocksDBStateBackend class, but I recorded the same result by using configuration parameter state.backend.incremental: true. Il giorno lun 1 lug 2019 alle ore 14:41 Yun Tang mailto:myas...@live.com><mailto:myas...@live.com<mailto:myas...@live.com>>> ha scritto: Hi Andrea The error happens when Flink try to verify whether your local backup directory existed[1]. If you could reproduce this, would you please share your configuration to RocksDBStateBackend, and what `fs.default-scheme` have you configured. Taskmanager log with more details is also very welcome. [1] https://github.com/apache/flink/blob/6f4148180ba372a2c12c1d54bea8579350af6c98/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L2568 Best Yun Tang From: Andrea Spina mailto:andrea.sp...@radicalbit.io><mailto:andrea.sp...@radicalbit.io<mailto:andrea.sp...@radicalbit.io>>> Sent: Monday, July 1, 2019 20:06 To: dev@flink.apache.org<mailto:dev@flink.apache.org><mailto:dev@flink.apache.org<mailto:dev@flink.apache.org>> Subject: Fwd: Flink 1.6.4 Issue on RocksDB incremental checkpoints and fs.default-scheme Dear community, I am running through the following issue. whenever I use rocksdb as state backend along with incremental checkpoints, I get the following error: *Caused by: java.lang.Exception: Could not materialize checkpoint 1 for operator Service Join SuperService (6/8).at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942) ... 6 moreCaused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateExceptionat java.util.concurrent.FutureTask.report(FutureTask.java:122)at java.util.concurrent.FutureTask.get(FutureTask.java:192)at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53) at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:47) at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853) ... 5 moreCaused by: java.lang.IllegalStateExceptionat org.apache.flink.util.Preconditions.checkState(Preconditions.java:179) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.runSnapshot(RocksDBKeyedStateBackend.java:2568) at java.util.concurrent.FutureTask.run(FutureTask.java:266)at org.
Re: [VOTE] Migrate to sponsored Travis account
I noticed that switching to a separate Travis account to run CI is actually impossible from what https://issues.apache.org/jira/browse/INFRA-18703 said. Hope another option from Chesnay to speed up the CI progress would work soon. Best Yun Tang From: Jark Wu Sent: Friday, July 5, 2019 10:34 To: dev Cc: priv...@flink.apache.org; Bowen Li Subject: Re: [VOTE] Migrate to sponsored Travis account +1 for the migration and great thanks to Chesnay and Bowen for pushing this! Cheers, Jark On Fri, 5 Jul 2019 at 09:34, Congxian Qiu wrote: > +1 for the migration. > > Best, > Congxian > > > Hequn Cheng 于2019年7月4日周四 下午9:42写道： > > > +1. > > > > And thanks a lot to Chesnay for pushing this. > > > > Best, Hequn > > > > On Thu, Jul 4, 2019 at 8:07 PM Chesnay Schepler > > wrote: > > > > > Note that the Flinkbot approach isn't that trivial either; we can't > > > _just_ trigger builds for a branch in the apache repo, but would first > > > have to clone the branch/pr into a separate repository (that is owned > by > > > the github account that the travis account would be tied to). > > > > > > One roadblock after the next showing up... > > > > > > On 04/07/2019 11:59, Chesnay Schepler wrote: > > > > Small update with mostly bad news: > > > > > > > > INFRA doesn't know whether it is possible, and referred my to Travis > > > > support. > > > > They did point out that it could be problematic in regards to > > > > read/write permissions for the repository. > > > > > > > > From my own findings /so far/ with a test repo/organization, it does > > > > not appear possible to configure the Travis account used for a > > > > specific repository. > > > > > > > > So yeah, if we go down this route we may have to pimp the Flinkbot to > > > > trigger builds through the Travis REST API. > > > > > > > > On 04/07/2019 10:46, Chesnay Schepler wrote: > > > >> I've raised a JIRA > > > >> <https://issues.apache.org/jira/browse/INFRA-18703>with INFRA to > > > >> inquire whether it would be possible to switch to a different Travis > > > >> account, and if so what steps would need to be taken. > > > >> We need a proper confirmation from INFRA since we are not in full > > > >> control of the flink repository (for example, we cannot access the > > > >> settings page). > > > >> > > > >> If this is indeed possible, Ververica is willing sponsor a Travis > > > >> account for the Flink project. > > > >> This would provide us with more than enough resources than we need. > > > >> > > > >> Since this makes the project more reliant on resources provided by > > > >> external companies I would like to vote on this. > > > >> > > > >> Please vote on this proposal, as follows: > > > >> [ ] +1, Approve the migration to a Ververica-sponsored Travis > > > >> account, provided that INFRA approves > > > >> [ ] -1, Do not approach the migration to a Ververica-sponsored > Travis > > > >> account > > > >> > > > >> The vote will be open for at least 24h, and until we have > > > >> confirmation from INFRA. The voting period may be shorter than the > > > >> usual 3 days since our current is effectively not working. > > > >> > > > >> On 04/07/2019 06:51, Bowen Li wrote: > > > >>> Re: > Are they using their own Travis CI pool, or did the switch to > > > >>> an entirely different CI service? > > > >>> > > > >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are > > > >>> currently moving away from ASF's Travis to their own in-house metal > > > >>> machines at [1] with custom CI application at [2]. They've seen > > > >>> significant improvement w.r.t both much higher performance and > > > >>> basically no resource waiting time, "night-and-day" difference > > > >>> quoting Wes. > > > >>> > > > >>> Re: > If we can just switch to our own Travis pool, just for our > > > >>> project, then this might be something we can do fairly quickly? > > > >>> > > > >>> I believe so, according to [3] and [4] > > > >
Re: A question about RocksDBListState and RocksDBMapState.
Hi kaka
You're correct, these comments are not correct for RocksDBMapState now, will
correct it with a hotfix.
Best
Yun Tang
From: kaka chen
Sent: Friday, July 5, 2019 15:05
To: dev@flink.apache.org
Subject: A question about RocksDBListState and RocksDBMapState.
Hi All,
I noticed RocksDBListState and RocksDBMapState have both the following
comments:
* {@link RocksDBStateBackend} must ensure that we set the
* {@link org.rocksdb.StringAppendOperator} on the column family that
we use for our state since
* we use the {@code merge()} call.
However, from the source code, I found only RocksDBListState use
rocksdb::merge() function which used StringAppendTestOperator in rocksdb to
merge the list entries when compaction or getting key. I don't know why
RocksDBMapState is related with these comments, could someone to explain
it, thanks.
Thanks,
Kaka
Re: [DISCUSS] ARM support for Flink
Hi Xiyuan Have you ever tried to release RocksDB on ARM like official doc[1] suggests? From our experience, cross-building for ARM did not work fine with Vagrant and we have to build rocksDB's binary file on ARM separately. As frocksdb [2] might not always maintained in Flink, I think we'd better support to release RocksDB-java with ARM officially. [1] https://github.com/facebook/rocksdb/blob/master/java/RELEASE.md [2] https://github.com/dataArtisans/frocksdb Best Yun Tang From: Xiyuan Wang Sent: Tuesday, July 9, 2019 10:52 To: dev@flink.apache.org Subject: Re: [DISCUSS] ARM support for Flink Thanks for your help. I built the frocksdb locally on ARM and all the related tests are passed now. Except some tests which can be fixed easily, it seems that both building and testing are ran well on ARM. Basing on my test, Is it possible to support Flink on ARM officailly? Seem the worklist is not too long. And I can help with the CI testing part. Need Flink team's idea. Thanks. Dian Fu 于2019年7月8日周一 上午10:23写道： > Hi Xiyuan, > > Thanks for bring the discussion. > > WRT the exception, it's because the native bundled in the rocksdb jar file > isn't compiled with cross platform support. You can refer [1] for how to > build rocksdb which has ARM platform. > > WRT ARM support, the rocksdb currently used in Flink is hosted in the > Ververica git [2], so it won't be difficult to make it support ARM. > However, I guess this git exists just for temporary [3], not because we > want to add much feature in rocksdb. > > [1] https://github.com/facebook/rocksdb/issues/678 < > https://github.com/facebook/rocksdb/issues/678> > [2] https://github.com/dataArtisans/frocksdb < > https://github.com/dataArtisans/frocksdb> > [3] https://issues.apache.org/jira/browse/FLINK-10471 < > https://issues.apache.org/jira/browse/FLINK-10471> > > Regards, > Dian > > > 在 2019年7月8日，上午9:17，Xiyuan Wang 写道： > > > > Hi Flink: > > Recently we meet a problem that we have to test and run Flink on ARM > > arch. While after searching Flink community, I didn’t find an official > ARM > > release version. > > > > Since Flink is made by Java and Scala language which can be ran > > cross-platform usually, I think Flink can be built and ran on ARM > directly > > as well. Then with my local test, Flink was built and deployed success as > > expected. But some tests were failed due to ARM arch. For example: > > > > 1. MemoryArchitectureTest.testArchitectureNotUnknown:34 Values should be > > different. Actual: UNKNOWN > > 2. [ERROR] > > > testIterator(org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest) > > Time elapsed: 0.234 s <<< ERROR! > > java.io.IOException: Could not load the native RocksDB library > > at > > > org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest.testIteratorHelper(RocksDBRocksStateKeysIteratorTest.java:90) > > at > > > org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest.testIterator(RocksDBRocksStateKeysIteratorTest.java:63) > > Caused by: java.lang.UnsatisfiedLinkError: > > > /tmp/rocksdb-lib-81ca7930b92af2cca143a050c0338d34/librocksdbjni-linux64.so: > > > /tmp/rocksdb-lib-81ca7930b92af2cca143a050c0338d34/librocksdbjni-linux64.so: > > cannot open shared object file: No such file or directory (Possible > cause: > > can't load AMD 64-bit .so on a AARCH64-bit platform) > > … > > > > Since the test isn’t passed totally, we are not sure if Flink 100% > > support ARM or not. Is it possible for Flink to support ARM release > > officially? I guess it may be not a very huge work basing on Java. I > notice > > that Flink now uses trivis-ci which is X86 only for build & test check. > Is > > it possible to add an ARM arch CI as well? It can be non-voting first. > Then > > we can keep monitoring and fixing ARM related error. One day it’s stable > > enough, we can remove the non-voting tag and create Flink ARM release. > > > > There is an open source CI community called OpenLab[1] which can provide > > CI function and ARM resource to Flink by free. I’m one of the OpenLab > > members. If Flink commun think ARM support is fine, I can keep helping > > Flink to build and maintain the ARM CI job. There is an POC for Flink > ARM > > build job made by me on OpenLab system[2] and a live demo which built and > > run on an ARM VM[3]. You can take a look first. > > > > Eager to get everybody’s feedback. Any question is welcome. > > > > Thanks. > > > > [1]: https://openlabtesting.org/ > > [2]: https://github.com/theopenlab/flink/pull/1 > > [3]: http://114.115.168.52:8081/#/overview > >
Re: [ANNOUNCE] Jiangjie (Becket) Qin has been added as a committer to the Flink project
Congratulations Becket! Well deserved. Best Yun Tang From: Robert Metzger Sent: Thursday, July 18, 2019 15:41 To: dev Subject: [ANNOUNCE] Jiangjie (Becket) Qin has been added as a committer to the Flink project Hi all, I'm excited to announce that Jiangjie (Becket) Qin just became a Flink committer! Congratulations Becket! Best, Robert (on behalf of the Flink PMC)
Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds
+1 sounds good, more people could be encouraged to involve in paying attention to failure commit. Best Yun Tang From: Becket Qin Sent: Monday, July 22, 2019 9:44 To: dev Subject: Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds +1. Sounds a good idea to me. On Sat, Jul 20, 2019 at 7:07 PM Dian Fu wrote: > Thanks Jark for the proposal, sounds reasonable for me. +1. This ML could > be used for all the build notifications including master and CRON jobs. > > > 在 2019年7月20日，下午2:55，Xu Forward 写道： > > > > +1 ,Thanks jark for the proposal. > > Best > > Forward > > > > Jark Wu 于2019年7月20日周六 下午12:14写道： > > > >> Hi all, > >> > >> As far as I know, currently, email notifications of Travis builds for > >> master branch are sent to the commit author when a build was just > broken or > >> still is broken. And there is no email notifications for CRON builds. > >> > >> Recently, we are suffering from compile errors for scala-2.12 and java-9 > >> which are only ran in CRON jobs. So I'm figuring out a way to get > >> notifications of CRON builds (or all builds) to quick fix compile errors > >> and failed cron tests. > >> > >> After reaching out to @Chesnay Schepler (thanks > for > >> the helping), I know that we are using a Slack channel to receive all > >> failed build notifications. However, IMO, email notification might be a > >> better way than Slack channel to encourage more people to pay attention > on > >> the builds. > >> > >> So I'm here to propose to setup a bui...@flink.apache.org mailing list > for > >> receiving build notifications. I also find that Beam has such mailing > list > >> too[1]. After we have such a mailing list, we can integrate it to travis > >> according to the travis doc[2]. > >> > >> What do you think? Do we need a formal vote for this? > >> > >> Best and thanks, > >> Jark > >> > >> [1]: https://beam.apache.org/community/contact-us/ > >> [2]: > >> > >> > https://docs.travis-ci.com/user/notifications/#configuring-email-notifications > >> > >> < > >> > https://docs.travis-ci.com/user/notifications/#configuring-email-notifications > >>> > >> > >> < > >> > https://docs.travis-ci.com/user/notifications/#configuring-email-notifications > >>> > >> > >
Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project
Congratulations Zhijiang, well deserved. Best From: Yingjie Cao Sent: Tuesday, July 23, 2019 10:23 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project Congratulations Zhijiang! yangtao.yt 于2019年7月23日周二 上午10:17写道： > Congrats, Zhejiang! > > Best, > Tao Yang > > > 在 2019年7月23日，上午9:46，boshu Zheng 写道： > > > > Congratulations Zhijiang > > > > 发自我的 iPhone > > > >> 在 2019年7月23日，上午12:55，Xuefu Z 写道： > >> > >> Congratulations, Zhijiang! > >> > >>> On Mon, Jul 22, 2019 at 7:42 AM Bo WANG > wrote: > >>> > >>> Congratulations Zhijiang! > >>> > >>> > >>> Best, > >>> > >>> Bo WANG > >>> > >>> > >>> On Mon, Jul 22, 2019 at 10:12 PM Robert Metzger > >>> wrote: > >>> > Hey all, > > We've added another committer to the Flink project: Zhijiang Wang. > > Congratulations Zhijiang! > > Best, > Robert > (on behalf of the Flink PMC) > > >>> > >> > >> > >> -- > >> Xuefu Zhang > >> > >> "In Honey We Trust!" > >
Re: [ANNOUNCE] Kete Young is now part of the Flink PMC
Congratulations Kurt, Well deserved. Best From: Xu Forward Sent: Tuesday, July 23, 2019 17:53 To: dev Subject: Re: [ANNOUNCE] Kete Young is now part of the Flink PMC Congratulations Kurt! best forward Jark Wu 于2019年7月23日周二 下午5:49写道： > Congratulations Kurt! Well deserved. > > Cheers, > Jark > > On Tue, 23 Jul 2019 at 17:43, LakeShen wrote: > > > Congratulations Kurt! > > > > Congxian Qiu 于2019年7月23日周二 下午5:37写道： > > > > > Congratulations Kurt! > > > Best, > > > Congxian > > > > > > > > > Dian Fu 于2019年7月23日周二 下午5:36写道： > > > > > > > Congrats, Kurt! > > > > > > > > > 在 2019年7月23日，下午5:33，Zili Chen 写道： > > > > > > > > > > Congratulations Kurt! > > > > > > > > > > Best, > > > > > tison. > > > > > > > > > > > > > > > JingsongLee 于2019年7月23日周二 > > 下午5:29写道： > > > > > > > > > >> Congratulations Kurt! > > > > >> > > > > >> Best, Jingsong Lee > > > > >> > > > > >> > > > > >> -- > > > > >> From:Robert Metzger > > > > >> Send Time:2019年7月23日(星期二) 17:24 > > > > >> To:dev > > > > >> Subject:[ANNOUNCE] Kete Young is now part of the Flink PMC > > > > >> > > > > >> Hi all, > > > > >> > > > > >> On behalf of the Flink PMC, I'm happy to announce that Kete Young > is > > > now > > > > >> part of the Apache Flink Project Management Committee (PMC). > > > > >> > > > > >> Kete has been a committer since February 2017, working a lot on > > Table > > > > API / > > > > >> SQL. He's currently co-managing the 1.9 release! Thanks a lot for > > your > > > > work > > > > >> for Flink! > > > > >> > > > > >> Congratulations & Welcome Kurt! > > > > >> > > > > >> Best, > > > > >> Robert > > > > >> > > > > > > > > > > > > > >
Re: Something wrong with travis?
I met this problem again at https://api.travis-ci.com/v3/job/220732163/log.txt . Is there any place we could ask for help to contact tarvis or any clues we could use to figure out this? Best Yun Tang From: Yun Tang Sent: Monday, June 24, 2019 14:22 To: dev@flink.apache.org ; Kurt Young Subject: Re: Something wrong with travis? Unfortunately, I met this problem again just now https://api.travis-ci.org/v3/job/549534496/log.txt (the build overview https://travis-ci.org/apache/flink/builds/549534489). For those non-committers, including me, we have to close-reopen the PR or push another commit to re-trigger the PR check🙁 Best Yun Tang From: Chesnay Schepler Sent: Wednesday, June 19, 2019 16:59 To: dev@flink.apache.org; Kurt Young Subject: Re: Something wrong with travis? Recent builds are passing again. On 18/06/2019 08:34, Kurt Young wrote: > Hi dev, > > I noticed that all the travis tests triggered by pull request are failed > with the same error: > > "Cached flink dir /home/travis/flink_cache/x/flink does not exist. > Exiting build." > > Anyone have a clue on what happened and how to fix this? > > Best, > Kurt >
Re: [ANNOUNCE] Hequn becomes a Flink committer
Congratulations Hequn. Best Yun Tang From: Rong Rong Sent: Thursday, August 8, 2019 0:41 Cc: dev ; user Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer Congratulations Hequn, well deserved! -- Rong On Wed, Aug 7, 2019 at 8:30 AM mailto:xingc...@gmail.com>> wrote: Congratulations, Hequn! From: Xintong Song mailto:tonysong...@gmail.com>> Sent: Wednesday, August 07, 2019 10:41 AM To: dev@flink.apache.org<mailto:dev@flink.apache.org> Cc: user mailto:u...@flink.apache.org>> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer Congratulations~! Thank you~ Xintong Song On Wed, Aug 7, 2019 at 4:00 PM vino yang mailto:yanghua1...@gmail.com>> wrote: Congratulations! highfei2...@126.com<mailto:highfei2...@126.com> mailto:highfei2...@126.com>> 于2019年8月7日周三 下午7:09写道： > Congrats Hequn! > > Best, > Jeff Yang > > > Original Message > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer > From: Piotr Nowojski > To: JingsongLee > CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun ,dev > ,user > > > Congratulations :) > > On 7 Aug 2019, at 12:09, JingsongLee > mailto:lzljs3620...@aliyun.com>> wrote: > > Congrats Hequn! > > Best, > Jingsong Lee > > -- > From:Biao Liu mailto:mmyy1...@gmail.com>> > Send Time:2019年8月7日(星期三) 12:05 > To:Zhu Zhu mailto:reed...@gmail.com>> > Cc:Zili Chen mailto:wander4...@gmail.com>>; Jeff Zhang > mailto:zjf...@gmail.com>>; Paul > Lam mailto:paullin3...@gmail.com>>; jincheng sun > mailto:sunjincheng...@gmail.com>>; dev > mailto:dev@flink.apache.org>>; user > mailto:u...@flink.apache.org>> > Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer > > Congrats Hequn! > > Thanks, > Biao /'bɪ.aʊ/ > > > > On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu > mailto:reed...@gmail.com>> wrote: > Congratulations to Hequn! > > Thanks, > Zhu Zhu > > Zili Chen mailto:wander4...@gmail.com>> 于2019年8月7日周三 > 下午5:16写道： > Congrats Hequn! > > Best, > tison. > > > Jeff Zhang mailto:zjf...@gmail.com>> 于2019年8月7日周三 下午5:14写道： > Congrats Hequn! > > Paul Lam mailto:paullin3...@gmail.com>> 于2019年8月7日周三 > 下午5:08写道： > Congrats Hequn! Well deserved! > > Best, > Paul Lam > > 在 2019年8月7日，16:28，jincheng sun > mailto:sunjincheng...@gmail.com>> 写道： > > Hi everyone, > > I'm very happy to announce that Hequn accepted the offer of the Flink PMC > to become a committer of the Flink project. > > Hequn has been contributing to Flink for many years, mainly working on > SQL/Table API features. He's also frequently helping out on the user > mailing lists and helping check/vote the release. > > Congratulations Hequn! > > Best, Jincheng > (on behalf of the Flink PMC) > > > > -- > Best Regards > > Jeff Zhang > > >
Re: Custom type serializer inside Flink state
Hi Ying What version of Flink are you using and please more exception stack. Moreover, what is the relationship between `MyProtoClass2` and `MyProtoClass1`? As far as I know, registering the Message class should not be the proper solution. For the 2nd question, you could refer to FLINK-11333 [1] for more information. CC @Tzu-Li (Gordon) Tai<mailto:tzuli...@apache.org> as he might provide more information about this. [1] https://issues.apache.org/jira/browse/FLINK-11333 Best Yun Tang From: Ying Xu Sent: Thursday, August 8, 2019 1:51 To: dev@flink.apache.org Subject: Custom type serializer inside Flink state Hi Flink community: Our Flink application sends different types of protobuf messages on-the-wire. Since protobuf cannot be handled by Flink type serializer, we had to register custom Kyro serializer: env.getConfig().*registerTypeWithKryoSerializer*(MyProtoClass1.class, ProtobufSerializer.class); We found registering each protobuf class is not a viable solution for schema evolution. Particularly, when adding/removing new messages we would encounter errors when restoring state backend: Caused by: org.apache.flink.util.FlinkException: Could not restore operator state backend for StreamSource_d63019dde9166d2672f543f01f344085_(13/32) from any of the 1 provided restore options. at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135) ... 5 common frames omitted Caused by: org.apache.flink.runtime.state.BackendBuildingException: Failed when trying to restore operator state backend at org.apache.flink.runtime.state.DefaultOperatorStateBackendBuilder.build(DefaultOperatorStateBackendBuilder.java:86) at org.apache.flink.runtime.state.filesystem.FsStateBackend.createOperatorStateBackend(FsStateBackend.java:504) at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$operatorStateBackend$0(StreamTaskStateInitializerImpl.java:246) ... 7 common frames omitted Caused by: java.lang.IllegalStateException: Missing value for the key 'proto$*MyProtoClass2*' at org.apache.flink.util.LinkedOptionalMap.unwrapOptionals(LinkedOptionalMap.java:190) We then switch to registering the default protobuf class for the super class of all proto -- Message.class , and this issue appears to go away. see.getConfig().*addDefaultKryoSerializer*(Message.class, ProtobufSerializer .class); Questions: 1) It seems custom Kyro serializers are registered with the Flink state backend. Can we confirm when using the default Kyro serializer, only the super class (e.g Message.class) is registered and no specific protobuf message is associated with state ? 2) Will proto ser/de supported by Flink type serializer in the future and is there any longer term roadmap for supporting state evolution for protobuf-type messages? Thanks a lot.
Re: [VOTE] Flink Project Bylaws
+1 (non-binding) But I have a minor question about "code change" action, for those "[hotfix]" github pull requests [1], the dev mailing list would not be notified currently. I think we should change the description of this action. [1] https://flink.apache.org/contributing/contribute-code.html#code-contribution-process Best Yun Tang From: JingsongLee Sent: Tuesday, August 13, 2019 23:56 To: dev Subject: Re: [VOTE] Flink Project Bylaws +1 (non-binding) Thanks Becket. I've learned a lot from current bylaws. Best, Jingsong Lee -- From:Yu Li Send Time:2019年8月13日(星期二) 17:48 To:dev Subject:Re: [VOTE] Flink Project Bylaws +1 (non-binding) Thanks for the efforts Becket! Best Regards, Yu On Tue, 13 Aug 2019 at 16:09, Xintong Song wrote: > +1 (non-binding) > > Thank you~ > > Xintong Song > > > > On Tue, Aug 13, 2019 at 1:48 PM Robert Metzger > wrote: > > > +1 (binding) > > > > On Tue, Aug 13, 2019 at 1:47 PM Becket Qin wrote: > > > > > Thanks everyone for voting. > > > > > > For those who have already voted, just want to bring this up to your > > > attention that there is a minor clarification to the bylaws wiki this > > > morning. The change is in bold format below: > > > > > > one +1 from a committer followed by a Lazy approval (not counting the > > vote > > > > of the contributor), moving to lazy majority if a -1 is received. > > > > > > > > > > > > > Note that this implies that committers can +1 their own commits and > merge > > > > right away. *However, the committe**rs should use their best > judgement > > to > > > > respect the components expertise and ongoing development plan.* > > > > > > > > > This addition does not really change anything the bylaws meant to set. > It > > > is simply a clarification. If anyone who have casted the vote objects, > > > please feel free to withdraw the vote. > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > > > > On Tue, Aug 13, 2019 at 1:29 PM Piotr Nowojski > > > wrote: > > > > > > > +1 > > > > > > > > > On 13 Aug 2019, at 13:22, vino yang wrote: > > > > > > > > > > +1 > > > > > > > > > > Tzu-Li (Gordon) Tai 于2019年8月13日周二 下午6:32写道： > > > > > > > > > >> +1 > > > > >> > > > > >> On Tue, Aug 13, 2019, 12:31 PM Hequn Cheng > > > > wrote: > > > > >> > > > > >>> +1 (non-binding) > > > > >>> > > > > >>> Thanks a lot for driving this! Good job. @Becket Qin < > > > > >> becket@gmail.com > > > > >>>> > > > > >>> > > > > >>> Best, Hequn > > > > >>> > > > > >>> On Tue, Aug 13, 2019 at 6:26 PM Stephan Ewen > > > wrote: > > > > >>> > > > > >>>> +1 > > > > >>>> > > > > >>>> On Tue, Aug 13, 2019 at 12:22 PM Maximilian Michels < > > m...@apache.org > > > > > > > > >>>> wrote: > > > > >>>> > > > > >>>>> +1 It's good that we formalize this. > > > > >>>>> > > > > >>>>> On 13.08.19 10:41, Fabian Hueske wrote: > > > > >>>>>> +1 for the proposed bylaws. > > > > >>>>>> Thanks for pushing this Becket! > > > > >>>>>> > > > > >>>>>> Cheers, Fabian > > > > >>>>>> > > > > >>>>>> Am Mo., 12. Aug. 2019 um 16:31 Uhr schrieb Robert Metzger < > > > > >>>>>> rmetz...@apache.org>: > > > > >>>>>> > > > > >>>>>>> I changed the permissions of the page. > > > > >>>>>>> > > > > >>>>>>> On Mon, Aug 12, 2019 at 4:21 PM Till Rohrmann < > > > > >>> trohrm...@apache.org> > > > > >>>>>>> wrote: > > > > >>>>>>> > > > > >>>>>>>> +1 for the proposal. Thanks a lot for driving this > discussion > > > > >&g
Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer
Congratulations Andrey. Best Yun Tang From: Xintong Song Sent: Wednesday, August 14, 2019 21:40 To: Oytun Tez Cc: Zili Chen ; Till Rohrmann ; dev ; user Subject: Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer Congratulations Andery~! Thank you~ Xintong Song On Wed, Aug 14, 2019 at 3:31 PM Oytun Tez mailto:oy...@motaword.com>> wrote: Congratulations Andrey! I am glad the Flink committer team is growing at such a pace! --- Oytun Tez M O T A W O R D The World's Fastest Human Translation Platform. oy...@motaword.com<mailto:oy...@motaword.com> ― www.motaword.com<http://www.motaword.com/> On Wed, Aug 14, 2019 at 9:29 AM Zili Chen mailto:wander4...@gmail.com>> wrote: Congratulations Andrey! Best, tison. Till Rohrmann mailto:trohrm...@apache.org>> 于2019年8月14日周三 下午9:26写道： Hi everyone, I'm very happy to announce that Andrey Zagrebin accepted the offer of the Flink PMC to become a committer of the Flink project. Andrey has been an active community member for more than 15 months. He has helped shaping numerous features such as State TTL, FRocksDB release, Shuffle service abstraction, FLIP-1, result partition management and various fixes/improvements. He's also frequently helping out on the user@f.a.o mailing lists. Congratulations Andrey! Best, Till (on behalf of the Flink PMC)
Re: [DISCUSS] FLIP-50: Spill-able Heap Keyed State Backend
Big +1 for this feature. Our customers including me, have ever met dilemma where we have to use window to aggregate events in applications like real-time monitoring. The larger of timer and window state, the poor performance of RocksDB. However, switching to use FsStateBackend would always make me feel fear about the OOM errors. Look forward for more powerful enrichment to state-backend, and help Flink to achieve better performance together. Best Yun Tang From: Stephan Ewen Sent: Thursday, August 15, 2019 23:07 To: dev Subject: Re: [DISCUSS] FLIP-50: Spill-able Heap Keyed State Backend +1 for this feature. I think this will be appreciated by users, as a way to use the HeapStateBackend with a safety-net against OOM errors. And having had major production exposure is great. >From the implementation plan, it looks like this exists purely in a new module and does not require any changes in other parts of Flink's code. Can you confirm that? Other that that, I have no further questions and we could proceed to vote on this FLIP, from my side. Best, Stephan On Tue, Aug 13, 2019 at 10:00 PM Yu Li wrote: > Sorry for forgetting to give the link of the FLIP, here it is: > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend > > Thanks! > > Best Regards, > Yu > > > On Tue, 13 Aug 2019 at 18:06, Yu Li wrote: > > > Hi All, > > > > We ever held a discussion about this feature before [1] but now opening > > another thread because after a second thought introducing a new backend > > instead of modifying the existing heap backend is a better option to > > prevent causing any regression or surprise to existing in-production > usage. > > And since introducing a new backend is relatively big change, we regard > it > > as a FLIP and need another discussion and voting process according to our > > newly drafted bylaw [2]. > > > > Please allow me to quote the brief description from the old thread [1] > for > > the convenience of those who noticed this feature for the first time: > > > > > > *HeapKeyedStateBackend is one of the two KeyedStateBackends in Flink, > > since state lives as Java objects on the heap in HeapKeyedStateBackend > and > > the de/serialization only happens during state snapshot and restore, it > > outperforms RocksDBKeyeStateBackend when all data could reside in > memory.**However, > > along with the advantage, HeapKeyedStateBackend also has its > shortcomings, > > and the most painful one is the difficulty to estimate the maximum heap > > size (Xmx) to set, and we will suffer from GC impact once the heap memory > > is not enough to hold all state data. There’re several (inevitable) > causes > > for such scenario, including (but not limited to):* > > > > > > > > ** Memory overhead of Java object representation (tens of times of the > > serialized data size).* Data flood caused by burst traffic.* Data > > accumulation caused by source malfunction.**To resolve this problem, we > > proposed a solution to support spilling state data to disk before heap > > memory is exhausted. We will monitor the heap usage and choose the > coldest > > data to spill, and reload them when heap memory is regained after data > > removing or TTL expiration, automatically. Furthermore, *to prevent > > causing unexpected regression to existing usage of HeapKeyedStateBackend, > > we plan to introduce a new SpillableHeapKeyedStateBackend and change it > to > > default in future if proven to be stable. > > > > Please let us know your point of the feature and any comment is > > welcomed/appreciated. Thanks. > > > > [1] https://s.apache.org/pxeif > > [2] > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026 > > > > Best Regards, > > Yu > > >
Re: CiBot Update
Thanks for @Chesnay Schepler<mailto:ches...@apache.org> for this really helpful command! I agreed with @Dian Fu<mailto:dian0511...@gmail.com> that we should include this in the "Bot commands". I just wanted to find the exact command but found nothing in the template and come here for the command line. Best Yun Tang From: Congxian Qiu Sent: Monday, August 26, 2019 20:08 To: dev@flink.apache.org Subject: Re: CiBot Update Thanks Chesnay for the nice work, it's very helpful Best, Congxian Terry Wang 于2019年8月26日周一 下午6:59写道： > Very helpful! Thanks Chesnay! > Best, > Terry Wang > > > > > 在 2019年8月23日，下午11:47，Ethan Li 写道： > > > > Thank you very much Chesnay! This is helpful > > > >> On Aug 23, 2019, at 2:58 AM, Chesnay Schepler > wrote: > >> > >> @Ethan Li The source for the CiBot is available here < > https://github.com/flink-ci/ci-bot/>. The implementation of this command > is tightly connected to how the CiBot works; but conceptually it looks at a > PR, finds the most recent build that ran, and uses the Travis REST API to > restart the build. > >> Additionally, it keeps track of which comments have been processed by > storing the comment ID in the CI report. > >> If you have further questions, feel free to ping me directly. > >> > >> @Dianfu I agree, we should include it somewhere in either the flinkbot > template or the CI report. > >> > >> On 23/08/2019 03:35, Dian Fu wrote: > >>> Thanks Chesnay for your great work! A very useful feature! > >>> > >>> Just one minor suggestion: It will be better if we could add this > command to the section "Bot commands" in the flinkbot template. > >>> > >>> Regards, > >>> Dian > >>> > >>>> 在 2019年8月23日，上午2:06，Ethan Li 写道： > >>>> > >>>> My question is specifically about implementation of "@flinkbot run > travis" > >>>> > >>>>> On Aug 22, 2019, at 1:06 PM, Ethan Li > wrote: > >>>>> > >>>>> Hi Chesnay, > >>>>> > >>>>> This is really nice feature! > >>>>> > >>>>> Can I ask how is this implemented? Do you have the related > Jira/PR/docs that I can take a look? I’d like to introduce it to another > project if applicable. Thank you very much! > >>>>> > >>>>> Best, > >>>>> Ethan > >>>>> > >>>>>> On Aug 22, 2019, at 8:34 AM, Biao Liu mmyy1...@gmail.com>> wrote: > >>>>>> > >>>>>> Thanks Chesnay a lot, > >>>>>> > >>>>>> I love this feature! > >>>>>> > >>>>>> Thanks, > >>>>>> Biao /'bɪ.aʊ/ > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Thu, 22 Aug 2019 at 20:55, Hequn Cheng <mailto:chenghe...@gmail.com>> wrote: > >>>>>> > >>>>>>> Cool, thanks Chesnay a lot for the improvement! > >>>>>>> > >>>>>>> Best, Hequn > >>>>>>> > >>>>>>> On Thu, Aug 22, 2019 at 5:02 PM Zhu Zhu <mailto:reed...@gmail.com>> wrote: > >>>>>>> > >>>>>>>> Thanks Chesnay for the CI improvement! > >>>>>>>> It is very helpful. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Zhu Zhu > >>>>>>>> > >>>>>>>> zhijiang wangzhijiang...@aliyun.com.invalid>> 于2019年8月22日周四 下午4:18写道： > >>>>>>>> > >>>>>>>>> It is really very convenient now. Valuable work, Chesnay! > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Zhijiang > >>>>>>>>> > -- > >>>>>>>>> From:Till Rohrmann trohrm...@apache.org>> > >>>>>>>>> Send Time:2019年8月22日(星期四) 10:13 > >>>>>>>>> To:dev mailto:dev@flink.apache.org>> > >>>>>>>>> Subject:Re: CiBot Update > >>>>>>>>> > >>>>>>>>> Thanks for the continuous work on the CiBot Chesnay! > >
Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink
Hi Yijie I can see that Pulsar becomes more and more popular recently and very glad to see more people willing to contribute to Flink ecosystem. Before any further discussion, would you please give some explanation of the relationship between this thread to current existing JIRAs of pulsar source [1] and sink [2] connector? Will the contribution contains part of those PRs or totally different implementation? [1] https://issues.apache.org/jira/browse/FLINK-9641 [2] https://issues.apache.org/jira/browse/FLINK-9168 Best Yun Tang From: Yijie Shen Sent: Tuesday, September 3, 2019 13:57 To: dev@flink.apache.org Subject: [DISCUSS] Contribute Pulsar Flink connector back to Flink Dear Flink Community! I would like to open the discussion of contributing Pulsar Flink connector [0] back to Flink. ## A brief introduction to Apache Pulsar Apache Pulsar[1] is a multi-tenant, high-performance distributed pub-sub messaging system. Pulsar includes multiple features such as native support for multiple clusters in a Pulsar instance, with seamless geo-replication of messages across clusters, very low publish and end-to-end latency, seamless scalability to over a million topics, and guaranteed message delivery with persistent message storage provided by Apache BookKeeper. Nowadays, Pulsar has been adopted by more and more companies[2]. ## The status of Pulsar Flink connector The Pulsar Flink connector we are planning to contribute is built upon Flink 1.9.0 and Pulsar 2.4.0. The main features are: - Pulsar as a streaming source with exactly-once guarantee. - Sink streaming results to Pulsar with at-least-once semantics. (We would update this to exactly-once as well when Pulsar gets all transaction features ready in its 2.5.0 version) - Build upon Flink new Table API Type system (FLIP-37[3]), and can automatically (de)serialize messages with the help of Pulsar schema. - Integrate with Flink new Catalog API (FLIP-30[4]), which enables the use of Pulsar topics as tables in Table API as well as SQL client. ## Reference [0] https://github.com/streamnative/pulsar-flink [1] https://pulsar.apache.org/ [2] https://pulsar.apache.org/en/powered-by/ [3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System [4] https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs Best, Yijie Shen
Re: instable checkpointing after migration to flink 1.8
Hi Bekir
From what I could see, there should be two main factors influencing your time
of sync execution checkpoint within that task.
1. Snapshot timers in heap to S3 [1] (network IO)
2. Creating local RocksDB checkpoint on disk [2] (disk IO)
For the first part, unfortunately, there is no log or metrics could detect how
long it takes.
For the second part, you could login the machine where running that task, and
find logs of RocksDB (default DB folder is
{io.tmp.dirs}/flink-io-xxx/job-xxx/db and the log file name is LOG). You could
check the interval of logs between "Started the snapshot process -- creating
snapshot in directory" and "Snapshot DONE" to know how long RocksDB takes to
create checkpoint on local disk.
If we configure "state.backend.rocksdb.timer-service.factory" to "ROCKSDB", we
could avoid the 1st part of time and this might be a solution to your problem.
But to be honest, the implementation of timer snapshot code almost stay the
same for Flink-1.6 and Flink-1.8 and should not be a regression.
[1]
https://github.com/apache/flink/blob/ba1cd43b6ed42070fd9013159a5b65adb18b3975/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L453
[2]
https://github.com/apache/flink/blob/ba1cd43b6ed42070fd9013159a5b65adb18b3975/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java#L249
Best
Yun Tang
From: Congxian Qiu
Sent: Thursday, September 5, 2019 10:38
To: Bekir Oguz
Cc: Stephan Ewen ; dev ; Niels
Alebregtse ; Vladislav Bakayev
Subject: Re: instable checkpointing after migration to flink 1.8
Another information from our private emails
there ALWAYS have Kafka AbstractCoordinator logs about lost connection to
Kafka at the same time we have the checkpoints confirmed. Bekir checked the
Kafka broker log, but did not find any interesting things there.
Best,
Congxian
Congxian Qiu 于2019年9月5日周四 上午10:26写道：
> Hi Bekir,
>
> If it is the storage place for timers, for RocksDBStateBackend, timers can
> be stored in Heap or RocksDB[1][2]
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#tuning-rocksdb
> [2]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html#rocksdb-state-backend-config-options
>
> Best,
> Congxian
>
>
> Bekir Oguz 于2019年9月4日周三 下午11:38写道：
>
>> Hi Stephan,
>> sorry for late response.
>> We indeed use timers inside a KeyedProcessFunction but the triggers of
>> the timers are kinda evenly distributed, so not causing a firing storm.
>> We have a custom ttl logic which is used by the deletion timer to decide
>> whether delete a record from inmemory state or not.
>> Can you maybe give some links to those changes in the RocksDB?
>>
>> Thanks in advance,
>> Bekir Oguz
>>
>> On Fri, 30 Aug 2019 at 14:56, Stephan Ewen wrote:
>>
>>> Hi all!
>>>
>>> A thought would be that this has something to do with timers. Does the
>>> task with that behavior use timers (windows, or process function)?
>>>
>>> If that is the case, some theories to check:
>>> - Could it be a timer firing storm coinciding with a checkpoint?
>>> Currently, that storm synchronously fires, checkpoints cannot preempt that,
>>> which should change in 1.10 with the new mailbox model.
>>> - Could the timer-async checkpointing changes have something to do
>>> with that? Does some of the usually small "preparation work" (happening
>>> synchronously) lead to an unwanted effect?
>>> - Are you using TTL for state in that operator?
>>> - There were some changes made to support timers in RocksDB recently.
>>> Could that have something to do with it?
>>>
>>> Best,
>>> Stephan
>>>
>>>
>>> On Fri, Aug 30, 2019 at 2:45 PM Congxian Qiu
>>> wrote:
>>>
>>>> CC flink dev mail list
>>>> Update for those who may be interested in this issue, we'are still
>>>> diagnosing this problem currently.
>>>>
>>>> Best,
>>>> Congxian
>>>>
>>>>
>>>> Congxian Qiu 于2019年8月29日周四 下午8:58写道：
>>>>
>>>> > Hi Bekir
>>>> >
>>>> > Currently, from what we have diagnosed, there is some task complete
>>>> its
>>>> > checkpoint too late (maybe 15 mins), but we checked the kafka broker
>>>> log
>>>> > and did not find any interesting things there. could we run another
>>>> job,
>>&
Re: [DISCUSS] Support customize state in customized KeyedStateBackend
Hi all First of all, I agreed with Yu that we should support to make state type pluginable. If we take a look at current Flink implementation. Users could implement their pluginable state backend to satisfy their own meets now. However, even users could define their own state descriptor, they cannot store the customized state within their state backend. The root cause behind this is that current StateBackendFactory could accept user defined state backend factory while StateFactory (within HeapKeyedStateBackend [1] and RocksDBKeyedStateBackend [2] ) cannot. If we agreed that we should leave the right of implementing customized state backend to users, it's naturally to agree that we should also leave the right of implementing customized states to users. [1] https://github.com/apache/flink/blob/576228651382db040aaa006cf9142f6568930cb1/flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java#L79 [2] https://github.com/apache/flink/blob/576228651382db040aaa006cf9142f6568930cb1/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L114 Best Yun Tang From: Yu Li Sent: Monday, September 9, 2019 2:24 To: dev Subject: Re: [DISCUSS] Support customize state in customized KeyedStateBackend Hi Shimin, Thanks for bring this discussion up. First of all, I'd like to confirm/clarify that this discussion is mainly about managed state with customized state descriptor rather than raw state, right? Asking because raw state was the very first thing came to my mind when seeing the title. And this is actually the first topic/question we need to discuss, that whether we should support user-defined state descriptor and still ask framework to manage the state life cycle. Personally I'm +1 on this since the "official" state (data-structure) types (currently mainly value, list and map) may not be optimized for customer case, but we'd better ask others' opinion. Secondly, if the result of the first question is "Yes", then it's truly a problem that "Although we can implement customized StateDescriptors for different kind of data structures, we do not really have access to such customized state in RichFunctions", and how to resolve it is the second topic/question to discuss. I've noticed your proposal of exposing "getParitionedState" method out in "RuntimeContext" and "KeyedStateStore" in JIRA (FLINK-14003), but IMO adding a specific interface like below is better than exposing the internal one: State getCustomizedState(StateDescriptor stateProperties); Finally, I think this is a user-facing and definitely worthwhile discussion, and requires a FLIP to document the conclusion and design/implementation (if any) down. What's your opinion? Thanks. Best Regards, Yu On Fri, 6 Sep 2019 at 13:27, shimin yang wrote: > Hi every, > > I would like to start a discussion on supporting customize state > in customized KeyedStateBackend. > > In Flink, users can customize KeyedStateBackend to support different type > of data store. Although we can implement customized StateDescriptors for > different kind of data structrues, we do not really have access to such > customized state in RichFunctions. > > I propose to add a getOtherState method in RuntimeContext and > DefaultKeyedStateStore which directly takes StateDescriptor as parameter to > allow user to get customized state. > > What do you think? > > Thanks. > > Best, > Shimin >
Re: instable checkpointing after migration to flink 1.8
Hi Bekir
Changing the timer factory from HEAP to ROCKSDB would certainly help reduce
your JVM heap usage. Since it would use RocksDB to store the timer state, you
might come across performance regression as we need to poll timers from RocksDB
instead of JVM heap.
From our experience, 20 million timers per task manager still acts a bit too
much, could you reduce your window size to reduce the timers per window? By the
way, timer coalescing [1] might be an idea to reduce timers. (This method could
only take effect when user register timer currently).
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#timer-coalescing
Best
Yun Tang
From: Bekir Oguz
Sent: Wednesday, September 11, 2019 19:39
To: Yun Tang
Cc: dev@flink.apache.org ; Stephan Ewen
; Niels Alebregtse ;
Vladislav Bakayev
Subject: Re: instable checkpointing after migration to flink 1.8
Hi Yun,
first of all, this reported problem looks like resolved for 2 days already,
right after we changed the type and number of our nodes to give more heap to
task managers and have more task-managers as well.
Previously, our job was configured as parallelism 70, which was distributed to
14 task-managers (5 slots/tm) each having 9GB heap.
After our config change, the job is now using parallelism 100, distributed to
20 task-managers (5 slots/tm) each having 18GB heap.
This job has a keyed process function which manages ~400 million records in its
state, and creating 1 timer per record scheduled to trigger in 6 months to
check if the record is eligible to be wiped out from state or not. So we have
400M records + 400M timers.
For user state, we use RocksDB backend with incremental checkpointing enabled
and using s3 as an external checkpoint location. RocksDB backend is configured
with predefined FLASH_SSD_OPTIMIZED values.
Before our config change, flink was managing 400M/14=28,5M records + 28,5M
timers in each task-manager with 9GB heap.
And after the config change, this is 400M/20=20M records + 20M timers in each
task-manager with 18GB heap.
So, we have less state to manage per task manager, and have more heap.
Apparently this fixes(!) the problem of long checkpointing durations (15
minutes) happening occasionally.
Coming back to your points:
1. Snapshot timers are indeed using HEAP which is the default. We can set it to
ROCKSDB to see if that change has an impact on the end-to-end checkpoint
duration. Do you think this change will also reduce the heap usage?
2. I have collected and shared those logs under /tmp directory earlier and
noticed that snapshotting happens very fast, finishing in a second. But what I
noticed was, compaction kicking in during the snapshotting phase of a long (15
minutes) checkpoint. But still, the time spent for snapshotting was 1 second. I
guess compaction has no impact there. And still do not know why it took 15 mins
to acknowledge for one task slot.
I have another question regarding this problem and our use of timers. Is this a
good practice to use timers like we do? Does the flink timer service support
having this many timers? One timer per record, which is 400 million for us.
Looks like our problem is solved for the time being, but may appear again since
we still do not know the root cause.
About the use of timers: Could you please share your opinion on our timer setup
and maybe support us on my question on switching timers to use rocksdb instead
of heap?
Thanks a lot,
Bekir Oguz
On Thu, 5 Sep 2019 at 19:55, Yun Tang
mailto:myas...@live.com>> wrote:
Hi Bekir
From what I could see, there should be two main factors influencing your time
of sync execution checkpoint within that task.
1. Snapshot timers in heap to S3 [1] (network IO)
2. Creating local RocksDB checkpoint on disk [2] (disk IO)
For the first part, unfortunately, there is no log or metrics could detect how
long it takes.
For the second part, you could login the machine where running that task, and
find logs of RocksDB (default DB folder is
{io.tmp.dirs}/flink-io-xxx/job-xxx/db and the log file name is LOG). You could
check the interval of logs between "Started the snapshot process -- creating
snapshot in directory" and "Snapshot DONE" to know how long RocksDB takes to
create checkpoint on local disk.
If we configure "state.backend.rocksdb.timer-service.factory" to "ROCKSDB", we
could avoid the 1st part of time and this might be a solution to your problem.
But to be honest, the implementation of timer snapshot code almost stay the
same for Flink-1.6 and Flink-1.8 and should not be a regression.
[1]
https://github.com/apache/flink/blob/ba1cd43b6ed42070fd9013159a5b65adb18b3975/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L453
[2]
https://github.com/apache/flink/blob/ba1cd43b6ed42070fd9013159a5b65adb18b3975/flink-stat
Re: [ANNOUNCE] Zili Chen becomes a Flink committer
Congratulations Zili Best Yun Tang From: Yun Gao Sent: Thursday, September 12, 2019 10:12 To: dev Subject: Re: [ANNOUNCE] Zili Chen becomes a Flink committer Congratulations Zili! Best, Yun -- From:Yangze Guo Send Time:2019 Sep. 12 (Thu.) 09:38 To:dev Subject:Re: [ANNOUNCE] Zili Chen becomes a Flink committer Congratulations! Best, Yangze Guo On Thu, Sep 12, 2019 at 9:35 AM Rong Rong wrote: > > Congratulations Zili! > > -- > Rong > > On Wed, Sep 11, 2019 at 6:26 PM Hequn Cheng wrote: > > > Congratulations! > > > > Best, Hequn > > > > On Thu, Sep 12, 2019 at 9:24 AM Jark Wu wrote: > > > >> Congratulations Zili! > >> > >> Best, > >> Jark > >> > >> On Wed, 11 Sep 2019 at 23:06, wrote: > >> > >> > Congratulations, Zili. > >> > > >> > > >> > > >> > Best, > >> > > >> > Xingcan > >> > > >> > > >> > > >> > *From:* SHI Xiaogang > >> > *Sent:* Wednesday, September 11, 2019 7:43 AM > >> > *To:* Guowei Ma > >> > *Cc:* Fabian Hueske ; Biao Liu ; > >> > Oytun Tez ; bupt_ljy ; dev < > >> > dev@flink.apache.org>; user ; Till Rohrmann < > >> > trohrm...@apache.org> > >> > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer > >> > > >> > > >> > > >> > Congratulations! > >> > > >> > > >> > > >> > Regards, > >> > > >> > Xiaogang > >> > > >> > > >> > > >> > Guowei Ma 于2019年9月11日周三 下午7:07写道： > >> > > >> > Congratulations Zili ! > >> > > >> > > >> > Best, > >> > > >> > Guowei > >> > > >> > > >> > > >> > > >> > > >> > Fabian Hueske 于2019年9月11日周三 下午7:02写道： > >> > > >> > Congrats Zili Chen :-) > >> > > >> > > >> > > >> > Cheers, Fabian > >> > > >> > > >> > > >> > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu < > >> mmyy1...@gmail.com>: > >> > > >> > Congrats Zili! > >> > > >> > > >> > > >> > Thanks, > >> > > >> > Biao /'bɪ.aʊ/ > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > On Wed, 11 Sep 2019 at 18:43, Oytun Tez wrote: > >> > > >> > Congratulations! > >> > > >> > > >> > > >> > --- > >> > > >> > Oytun Tez > >> > > >> > > >> > > >> > *M O T A W O R D* > >> > > >> > *The World's Fastest Human Translation Platform.* > >> > > >> > oy...@motaword.com — www.motaword.com<http://www.motaword.com> > >> > > >> > > >> > > >> > > >> > > >> > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy wrote: > >> > > >> > Congratulations! > >> > > >> > > >> > > >> > Best, > >> > > >> > Jiayi Liao > >> > > >> > > >> > > >> > Original Message > >> > > >> > *Sender:* Till Rohrmann > >> > > >> > *Recipient:* dev; user > >> > > >> > *Date:* Wednesday, Sep 11, 2019 17:22 > >> > > >> > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer > >> > > >> > > >> > > >> > Hi everyone, > >> > > >> > > >> > > >> > I'm very happy to announce that Zili Chen (some of you might also know > >> > him as Tison Kun) accepted the offer of the Flink PMC to become a > >> committer > >> > of the Flink project. > >> > > >> > > >> > > >> > Zili Chen has been an active community member for almost 16 months now. > >> > He helped pushing the Flip-6 effort over the finish line, ported a lot > >> of > >> > legacy code tests, removed a good part of the legacy code, contributed > >> > numerous fixes, is involved in the Flink's client API refactoring, > >> drives > >> > the refactoring of Flink's HighAvailabilityServices and much more. Zili > >> > Chen also helped the community by PR reviews, reporting Flink issues, > >> > answering user mails and being very active on the dev mailing list. > >> > > >> > > >> > > >> > Congratulations Zili Chen! > >> > > >> > > >> > > >> > Best, Till > >> > > >> > (on behalf of the Flink PMC) > >> > > >> > > >> > >
Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren
Congratulations, Qingsheng! Best Yun Tang From: weijie guo Sent: Sunday, April 23, 2023 14:50 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren Congratulations, Qingsheng! Best regards, Weijie Geng Biao 于2023年4月23日周日 14:29写道： > Congrats, Qingsheng! > Best, > Biao Geng > > 获取 Outlook for iOS<https://aka.ms/o0ukef> > > 发件人: Wencong Liu > 发送时间: Sunday, April 23, 2023 11:06:39 AM > 收件人: dev@flink.apache.org > 主题: Re:[ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren > > Congratulations, Qingsheng! > > Best, > Wencong LIu > > > > > > > > > > > > > > > > At 2023-04-21 19:47:52, "Jark Wu" wrote: > >Hi everyone, > > > >We are thrilled to announce that Leonard Xu has joined the Flink PMC! > > > >Leonard has been an active member of the Apache Flink community for many > >years and became a committer in Nov 2021. He has been involved in various > >areas of the project, from code contributions to community building. His > >contributions are mainly focused on Flink SQL and connectors, especially > >leading the flink-cdc-connectors project to receive 3.8+K GitHub stars. He > >authored 150+ PRs, and reviewed 250+ PRs, and drove several FLIPs (e.g., > >FLIP-132, FLIP-162). He has participated in plenty of discussions in the > >dev mailing list, answering questions about 500+ threads in the > >user/user-zh mailing list. Besides that, he is community minded, such as > >being the release manager of 1.17, verifying releases, managing release > >syncs, etc. > > > >Congratulations and welcome Leonard! > > > >Best, > >Jark (on behalf of the Flink PMC) > > > > > > > > > > > > > > > At 2023-04-21 19:50:02, "Jark Wu" wrote: > >Hi everyone, > > > >We are thrilled to announce that Qingsheng Ren has joined the Flink PMC! > > > >Qingsheng has been contributing to Apache Flink for a long time. He is the > >core contributor and maintainer of the Kafka connector and > >flink-cdc-connectors, bringing users stability and ease of use in both > >projects. He drove discussions and implementations in FLIP-221, FLIP-288, > >and the connector testing framework. He is continuously helping with the > >expansion of the Flink community and has given several talks about Flink > >connectors at many conferences, such as Flink Forward Global and Flink > >Forward Asia. Besides that, he is willing to help a lot in the community > >work, such as being the release manager for both 1.17 and 1.18, verifying > >releases, and answering questions on the mailing list. > > > >Congratulations and welcome Qingsheng! > > > >Best, > >Jark (on behalf of the Flink PMC) >
Re: [ANNOUNCE] New Apache Flink PMC Member - Leonard Xu
Congratulations, Leonard! Best, Yun Tang From: weijie guo Sent: Sunday, April 23, 2023 14:50 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Leonard Xu Congratulations, Leonard! Best regards, Weijie Geng Biao 于2023年4月23日周日 14:30写道： > Congrats, Lenorad! > Best, > Biao Geng > > 获取 Outlook for iOS<https://aka.ms/o0ukef> > > 发件人: Wencong Liu > 发送时间: Sunday, April 23, 2023 11:05:41 AM > 收件人: dev@flink.apache.org > 主题: Re:[ANNOUNCE] New Apache Flink PMC Member - Leonard Xu > > Congratulations, Leonard! > > Best, > Wencong LIu > > > > > > > > > > > > > > > > At 2023-04-21 19:47:52, "Jark Wu" wrote: > >Hi everyone, > > > >We are thrilled to announce that Leonard Xu has joined the Flink PMC! > > > >Leonard has been an active member of the Apache Flink community for many > >years and became a committer in Nov 2021. He has been involved in various > >areas of the project, from code contributions to community building. His > >contributions are mainly focused on Flink SQL and connectors, especially > >leading the flink-cdc-connectors project to receive 3.8+K GitHub stars. He > >authored 150+ PRs, and reviewed 250+ PRs, and drove several FLIPs (e.g., > >FLIP-132, FLIP-162). He has participated in plenty of discussions in the > >dev mailing list, answering questions about 500+ threads in the > >user/user-zh mailing list. Besides that, he is community minded, such as > >being the release manager of 1.17, verifying releases, managing release > >syncs, etc. > > > >Congratulations and welcome Leonard! > > > >Best, > >Jark (on behalf of the Flink PMC) >
Re: [SUMMARY] Flink 1.18 Release Sync 04/18/2023
Hi Qingsheng, Thanks for sharing the sync summary. Since most developers in China would have a 5-day Labor Day Holiday on May 2nd, do we consider changing the sync meeting date after May 4th?<https://cn.bing.com/dict/search?q=Holiday&FORM=BDVSP6&cc=cn> Best Yun Tang From: Jing Ge Sent: Tuesday, April 25, 2023 4:30 To: Qingsheng Ren Cc: dev ; Konstantin Knauf ; snuyan...@gmail.com Subject: Re: [SUMMARY] Flink 1.18 Release Sync 04/18/2023 Hi Qingsheng, Thanks for sharing the summary! Best regards, Jing On Mon, Apr 24, 2023 at 1:50 PM Qingsheng Ren wrote: > Hi devs, > > I'd like to share some highlights in the 1.18 release sync on 04/18/2023 > (Sorry for the late summary!): > > - Feature list: @contributors please add your features to the list in > release 1.18 wiki page [1] so that we could track the overall progress. > - CI instabilities: owners of issues have already been pinged. > - Version management: as 1.17 has already been released, 1.15 related > resources like CIs and docker images will be removed in the coming week. > > The next release sync will be on May 2nd, 2023. Feel free and welcome to > join us[2] ! > > [1] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release > [2] > https://us04web.zoom.us/j/79158702091?pwd=8CXPqxMzbabWkma5b0qFXI1IcLbxBh.1 > > Best regards, > Jing, Konstantin, Sergey and Qingsheng >
Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints
+1 (binding) Thanks Zakelly for driving this topic. Best Yun Tang From: Yu Li Sent: Wednesday, May 10, 2023 15:44 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints +1 (binding) Thanks Zakelly for driving this, and thanks everyone for the thorough discussion. On Wed, May 10, 2023 at 11:15 AM Rui Fan <1996fan...@gmail.com> wrote: > Thanks for driving this proposal, Zakelly. > > +1(binding) > > Best, > Rui Fan > > On Wed, May 10, 2023 at 11:04 AM Hangxiang Yu wrote: > > > Hi Zakelly. > > Thanks for driving this. > > +1 (no-binding) > > > > On Wed, May 10, 2023 at 10:52 AM Yuan Mei > wrote: > > > > > Thanks for driving this, Zakelly. > > > > > > As discussed in the thread, > > > > > > +1 for the proposal (binding) > > > > > > Best, > > > > > > Yuan > > > > > > > > > > > > On Wed, May 10, 2023 at 10:39 AM Zakelly Lan > > > wrote: > > > > > > > Hi everyone, > > > > > > > > Sorry for the 4 duplicate emails. There was a problem with the dev > > > > mailing list blocking the mails from Gmail. I thought it was a > network > > > > problem so I tried several times. The issue is addressed by > > > > INFRA-24572[1] and the piled up mails are delivered at one time. > > > > > > > > Based on the sending time, the vote will be open until May 12th at > > > > 11:00PM GMT. Please discuss and vote in the last thread (this one). > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > Zakelly > > > > > > > > [1] https://issues.apache.org/jira/browse/INFRA-24572 > > > > > > > > On Wed, May 10, 2023 at 10:30 AM Yanfei Lei > > wrote: > > > > > > > > > > +1 (no-binding) > > > > > > > > > > Best, > > > > > Yanfei > > > > > > > > > > > > > > > Jing Ge 于2023年5月10日周三 07:03写道： > > > > > > > > > > > > > > > > > Hi Zakelly, > > > > > > > > > > > > I saw you sent at least 4 same emails for voting FLIP-306. I > guess > > > > this one > > > > > > should be the last one and the right one for us to vote right? > BTW, > > > > based > > > > > > on the sending time, 72 hours means to open the discussion until > > May > > > > 12th. > > > > > > > > > > > > Best regards, > > > > > > Jing > > > > > > > > > > > > On Tue, May 9, 2023 at 8:24 PM Zakelly Lan < > zakelly@gmail.com> > > > > wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > Thanks for all the feedback for FLIP-306: Unified File Merging > > > > > > > Mechanism for Checkpoints[1] on the discussion thread[2]. > > > > > > > > > > > > > > I'd like to start a vote for it. The vote will be open for at > > least > > > > 72 > > > > > > > hours (until May 11th, 12:00AM GMT) unless there is an > objection > > or > > > > an > > > > > > > insufficient number of votes. > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Zakelly > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints > > > > > > > [2] > > > https://lists.apache.org/thread/56px3kvn3tlwpc7sl12kx6notfmk9g7q > > > > > > > > > > > > > > > > > > > > -- > > Best, > > Hangxiang. > > > -- Best Regards, Yu
Re: [DISCUSS] Release Flink 1.17.1
+1 for release flink-1.17.1 The blocker issue might cause silent incorrect data, it's better to have a fix release ASAP. Best Yun Tang From: weijie guo Sent: Thursday, May 11, 2023 11:08 To: dev@flink.apache.org ; tonysong...@gmail.com Subject: [DISCUSS] Release Flink 1.17.1 Hi all, I would like to discuss creating a new 1.17 patch release (1.17.1). The last 1.17 release is nearly two months old, and since then, 66 tickets have been closed [1], of which 14 are blocker/critical [2]. Some of them are quite important, such as FLINK-31293 [3] and FLINK-32027 [4]. I am not aware of any unresolved blockers and there are no in-progress tickets [5]. Please let me know if there are any issues you'd like to be included in this release but still not merged. If the community agrees to create this new patch release, I could volunteer as the release manager and Xintong can help with actions that require a PMC role. Thanks, Weijie [1] https://issues.apache.org/jira/browse/FLINK-32027?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.1%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC [2] https://issues.apache.org/jira/browse/FLINK-31273?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.1%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC [3] https://issues.apache.org/jira/browse/FLINK-31293 [4] https://issues.apache.org/jira/browse/FLINK-32027 [5] https://issues.apache.org/jira/projects/FLINK/versions/12352886
Re: [DISCUSS] FLIP-229: Prometheus Sink Connector
+1 for the idea to introduce the prometheus connector. In some cases, users cannot leverage the current way to report very large amounts of metrics via metrics reporter due to the limit on the message collection of prometheus agent. For the FLIP itself, I think we should also consider the rate limit feature to avoid breaking down the metrics services. Best Yun Tang From: Jing Ge Sent: Friday, May 19, 2023 4:49 To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-229: Prometheus Sink Connector Hi Karthi, Thanks for raising this proposal. It is a common use case to sink metric "data" into downstream Prometheus. The description in the motivation section is more or less misleading the discussion. I would suggest you rephrase it, e.g. metrics (pre)processing via Flink is The current Flip does not contain too much information about how data will be sinked into Prometheus. Would you like to elaborate on it? Some architecture diagrams might help us better understand your thoughts. Just out of curiosity, since you are focusing on building the Prometheus connector, does it make sense to build the Prometheus Source too? Short-term stored metrics could be consumed by the Prometheus Source and, depending on requirements, might perform some processing like aggregation. Best regards, Jing On Wed, May 17, 2023 at 6:13 PM Danny Cranmer wrote: > Thanks for the FLIP. > > I agree that there is a real usecase for metrics sink vs metric reporter. > The metric reporters in Flink cover metrics about the Flink job, and a sink > is used when the metrics are the _data_. > > +1 on the FLIP ID, can you please fix that? > > With regard to this FLIP. > 1/ The fact we are using the remote write feature is not covered beyond the > code example. Can we add details on this to make it clear? Additionally > would this support _any_ Prometheus server or do we need to enable remote > endpoint feature on ther server? > 2/ Are there any concerns around Prometheus versioning or is the API > backwards compatible? Which versions of Prometheus will we be supporting? > 3/ With regard to the "AmazonPrometheusRequestSigner" the example has > static creds. Can we integrate with the AWS Util to support all credential > providers, static and dynamic? > > Thanks, > Danny > > On Wed, May 17, 2023 at 4:34 PM Teoh, Hong > wrote: > > > Thanks Karthi for creating the FLIP! > > > > Re: Martijn, > > > > As I understand it, the motivation for the Prometheus Sink is for users > > who want to write metrics to a Prometheus remote_write endpoint as an > > output of their job graph, so it would be good to treat it as a > first-class > > citizen and do it as part of Flink’s data flow. This way, we would > benefit > > from at least once guarantees, scaling, state management. > > > > For example, a user might want to read in logs, perform some aggregations > > and publish it into a metrics store for visualisation. This might be a > > great use-case for reducing the cardinality of metrics! > > > > I think you might be referring to the metrics of the Flink job itself > > (e.g. CPU / memory metrics). For this use case, I would agree that we > > should not use this sink but instead use our metric reporters. > > > > Regards, > > Hong > > > > > > > > > On 16 May 2023, at 03:37, Lijie Wang wrote: > > > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > > > > > Hi Karthi, > > > > > > I think you are using a wrong FLIP id, the FLIP-229 has already be > > used[1]. > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job > > > > > > Best, > > > Lijie > > > > > > Martijn Visser 于2023年5月16日周二 04:44写道： > > > > > >> Hi Karthi, > > >> > > >> Thanks for the FLIP and opening up the discussion. My main question > is: > > why > > >> should we create a separate connector and not use and/or improve the > > >> existing integrations with Prometheus? I would like to understand more > > so > > >> that it can be added to the motivation of the FLIP. > > >> > > >> Best regards, > > >> > > >> Martijn > > >> > > >> On Mon, May 15, 2023 at 6:03 PM Karthi Thyagarajan < > > kar...@karthitect.com> > > >> wrote: > > >> > > >>> Hello all, > > >>> > > >>> We would like to start a discussion thread on FLIP-229: Prometheus > Sink > > >>> Connector [1] where we propose to provide a sink connector for > > Prometheus > > >>> [2] based on the Async Sink [3]. Looking forward to comments and > > >> feedback. > > >>> Thank you. > > >>> > > >>> [1] > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Prometheus+Sink+Connector > > >>> [2] https://prometheus.io/ > > >>> [3] > > >>> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink > > >>> > > >> > > > > >
Re: [VOTE] Release 1.16.2, release candidate #1
+1 (non-binding) * Verified signatures * Reviewed the flink-web PR * Set up a standalone cluster from released binaries and check the git revision number. * Submit the statemachine example with RocksDB, and it works fine. Best, Yun Tang From: Yuxin Tan Sent: Friday, May 19, 2023 17:41 To: dev@flink.apache.org Subject: Re: [VOTE] Release 1.16.2, release candidate #1 +1 (non-binding) - Verified signature - Verified hashes - Build form source with mac - Verify that the source archives do not contain any binaries - Run streaming and batch job in sql-client successfully. Thanks weijie for driving this release candidate. Best, Yuxin weijie guo 于2023年5月19日周五 16:19写道： > Hi everyone, > > > Please review and vote on the release candidate #1 for the version 1.16.2, > > as follows: > > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source release and binary convenience releases to be > > deployed to dist.apache.org [2], which are signed with the key with > > fingerprint 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tag "release-1.16.2-rc1" [5], > > * website pull request listing the new release and adding announcement blog > > post [6]. > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > Thanks, > > Release Manager > > > [1] > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352765 > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.16.2-rc1/ > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > https://repository.apache.org/content/repositories/orgapacheflink-1634/ > > [5] https://github.com/apache/flink/releases/tag/release-1.16.2-rc1 > > [6] https://github.com/apache/flink-web/pull/649 >
Re: [VOTE] Release 1.17.1, release candidate #1
+1 (non-binding) * Verified signatures. * Start a local standalone cluster and checked the git revision number. * Submit stateful examples both in BATCH and STREAMING mode successfully. * Run a local sql-gateway. * Reviewed the release flink-web PR. Best Yun Tang From: Xingbo Huang Sent: Monday, May 22, 2023 19:10 To: dev@flink.apache.org Subject: Re: [VOTE] Release 1.17.1, release candidate #1 +1 (binding) - verify signatures and checksums - verify python wheel package contents - pip install apache-flink-libraries and apache-flink wheel packages - run example flink/flink-python/pyflink/examples/table/basic_operations.py with Python 3.10 - reviewed release blog post Best, Xingbo Yuxin Tan 于2023年5月22日周一 18:55写道： > +1 (non-binding) > > - verified the signatures > - verified the checksums > - built from source > - start a standalone cluster, run a simple batch and a streaming job > successfully > - review release announcement PR > > Best, > Yuxin > > > Xintong Song 于2023年5月22日周一 18:24写道： > > > +1 (binding) > > > > - verified signatures and checksums > > - built from source > > - tried example jobs with a standalone cluster, everything seems fine > > - review release announcement PR > > > > Best, > > > > Xintong > > > > > > > > On Mon, May 22, 2023 at 2:18 PM weijie guo > > wrote: > > > > > Hi everyone, > > > > > > > > > Please review and vote on the release candidate #1 for the version > > 1.17.1, > > > > > > as follows: > > > > > > [ ] +1, Approve the release > > > > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > > > > The complete staging area is available for your review, which includes: > > > > > > > > > * JIRA release notes [1], > > > > > > * the official Apache source release and binary convenience releases to > > be > > > > > > deployed to dist.apache.org [2], which are signed with the key with > > > > > > fingerprint 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [3], > > > > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > > > > * source code tag "release-1.17.1-rc1" [5], > > > > > > * website pull request listing the new release and adding announcement > > blog > > > > > > post [6]. > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > Thanks, > > > > > > Release Manager > > > > > > > > > [1] > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352886 > > > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.17.1-rc1/ > > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > > > > [4] > > > > https://repository.apache.org/content/repositories/orgapacheflink-1635/ > > > > > > [5] https://github.com/apache/flink/releases/tag/release-1.17.1-rc1 > > > > > > [6] https://github.com/apache/flink-web/pull/650 > > > > > >
Re: How to figure out what's the size of ListState?
Hi Amir, For the current Flink, you have to iterator the returned Iterable of ListState#get(). Why Flink lacks an API to get the size of listState directly? This is because Flink leverages RocksDB's merge operator [1] for high-performance listState#add [2], that is to say, we would not even read the original data when appending the listState. I think you can record the size in another valueState when implementing your logic. [1] https://github.com/facebook/rocksdb/wiki/Merge-Operator [2] https://github.com/apache/flink/blob/d8c64a808484cab78c8bd7b74a287edf7d1f3b01/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBListState.java#L131 Best Yun Tang From: Amir Hossein Sharifzadeh Sent: Tuesday, May 23, 2023 23:51 To: dev Subject: How to figure out what's the size of ListState? Dear Flink Dev team, It's about a while since I am dealing with an issue that can't figure out myself. I spent quite a lot of time trying to solve the problem myself, but I feel stuck. I explained the problem statement and the issue here: https://stackoverflow.com/questions/76308686/how-to-figure-out-whats-the-size-of-liststate I really appreciate any suggestion. Best, Amir
Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL
Hi Ron, I think this FLIP would help to improve the performance, looking forward to its completion in Flink! For the state compatibility session, it seems that the checkpoint compatibility would be broken just like [1] did. Could FLIP-190 [2] still be helpful in this case for SQL version upgrades? [1] https://docs.google.com/document/d/1qKVohV12qn-bM51cBZ8Hcgp31ntwClxjoiNBUOqVHsI/edit#heading=h.fri5rtpte0si [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489 Best Yun Tang From: Lincoln Lee Sent: Monday, June 5, 2023 10:56 To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL Hi Ron OFGC looks like an exciting optimization, looking forward to its completion in Flink! A small question, do we consider adding a benchmark for the operators to intuitively understand the improvement brought by each improvement? In addition, for the implementation plan, mentioned in the FLIP that 1.18 will support Calc, HashJoin and HashAgg, then what will be the next step? and which operators do we ultimately expect to cover (all or specific ones)? Best, Lincoln Lee liu ron 于2023年6月5日周一 09:40写道： > Hi, Jark > > Thanks for your feedback, according to my initial assessment, the work > effort is relatively large. > > Moreover, I will add a test result of all queries to the FLIP. > > Best, > Ron > > Jark Wu 于2023年6月1日周四 20:45写道： > > > Hi Ron, > > > > Thanks a lot for the great proposal. The FLIP looks good to me in > general. > > It looks like not an easy work but the performance sounds promising. So I > > think it's worth doing. > > > > Besides, if there is a complete test graph with all TPC-DS queries, the > > effect of this FLIP will be more intuitive. > > > > Best, > > Jark > > > > > > > > On Wed, 31 May 2023 at 14:27, liu ron wrote: > > > > > Hi, Jinsong > > > > > > Thanks for your valuable suggestions. > > > > > > Best, > > > Ron > > > > > > Jingsong Li 于2023年5月30日周二 13:22写道： > > > > > > > Thanks Ron for your information. > > > > > > > > I suggest that it can be written in the Motivation of FLIP. > > > > > > > > Best, > > > > Jingsong > > > > > > > > On Tue, May 30, 2023 at 9:57 AM liu ron wrote: > > > > > > > > > > Hi, Jingsong > > > > > > > > > > Thanks for your review. We have tested it in TPC-DS case, and got a > > 12% > > > > > gain overall when only supporting only Calc&HashJoin&HashAgg > > operator. > > > In > > > > > some queries, we even get more than 30% gain, it looks like an > > > effective > > > > > way. > > > > > > > > > > Best, > > > > > Ron > > > > > > > > > > Jingsong Li 于2023年5月29日周一 14:33写道： > > > > > > > > > > > Thanks Ron for the proposal. > > > > > > > > > > > > Do you have some benchmark results for the performance > > improvement? I > > > > > > am more concerned about the improvement on Flink than the data in > > > > > > other papers. > > > > > > > > > > > > Best, > > > > > > Jingsong > > > > > > > > > > > > On Mon, May 29, 2023 at 2:16 PM liu ron > > wrote: > > > > > > > > > > > > > > Hi, dev > > > > > > > > > > > > > > I'd like to start a discussion about FLIP-315: Support Operator > > > > Fusion > > > > > > > Codegen for Flink SQL[1] > > > > > > > > > > > > > > As main memory grows, query performance is more and more > > determined > > > > by > > > > > > the > > > > > > > raw CPU costs of query processing itself, this is due to the > > query > > > > > > > processing techniques based on interpreted execution shows poor > > > > > > performance > > > > > > > on modern CPUs due to lack of locality and frequent instruction > > > > > > > mis-prediction. Therefore, the industry is also researching how > > to > > > > > > improve > > > > > > > engine performance by increasing operator execution efficiency. > > In > > > > > > > addition, during the process of optimizing Flink's performance > > for > > > > TPC-DS > > > > > > > queries, we found that a significant amount of CPU time was > spent > > > on > > > > > > > virtual function calls, framework collector calls, and invalid > > > > > > > calculations, which can be optimized to improve the overall > > engine > > > > > > > performance. After some investigation, we found Operator Fusion > > > > Codegen > > > > > > > which is proposed by Thomas Neumann in the paper[2] can address > > > these > > > > > > > problems. I have finished a PoC[3] to verify its feasibility > and > > > > > > validity. > > > > > > > > > > > > > > Looking forward to your feedback. > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-315+Support+Operator+Fusion+Codegen+for+Flink+SQL > > > > > > > [2]: http://www.vldb.org/pvldb/vol4/p539-neumann.pdf > > > > > > > [3]: https://github.com/lsyldliu/flink/tree/OFCG > > > > > > > > > > > > > > Best, > > > > > > > Ron > > > > > > > > > > > > > > > >
Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode
Hi Feng, I think this FLIP would provide one important feature to unify the stream-SQL and batch-SQL when we backfill the historical data in batch mode. For the "Syntax" session, I think you could add descriptions of how to align backfill time travel with querying the latest data. And I think you should also update the "Discussion thread" in the original FLIP. Moreover, I have a question about getting the table schema from the catalog. I'm not sure whether the Catalog#getTable(tablePath, timestamp) will be called only once. If we have a backfill query between 2023-05-29 and 2023-06-04 in the past week, and the table schema changed on 2023-06-01, will the query below detect the schema changes during backfill the whole week? SELECT * FROM paimon_tb FOR SYSTEM_TIME AS OF TIMESTAMP BETWEEN '2023-05-29 00:00:00' AND '2023-06-05 00:00:00' Best Yun Tang From: Shammon FY Sent: Thursday, June 1, 2023 17:57 To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode Hi Feng, I have one minor comment about the public interface `Optional getSnapshot()` in the `CatalogTable`. As we can get tables from the new method `Catalog.getTable(ObjectPath tablePath, long timestamp)`, I think the returned `CatalogBaseTable` will have the information of timestamp. Flink or connector such as iceberg/paimon can create sources from the `CatalogBaseTable` directly without the need to get the snapshot ID from `CatalogTable.getSnapshot()`. What do you think of it? Best, Shammon FY On Thu, Jun 1, 2023 at 7:22 AM Jing Ge wrote: > Hi Feng, > > Thanks for the proposal! Very interesting feature. Would you like to update > your thoughts described in your previous email about why SupportsTimeTravel > has been rejected into the FLIP? This will help readers understand the > context (in the future). > > Since we always directly add overload methods into Catalog according to new > requirements, which makes the interface bloated. Just out of curiosity, > does it make sense to introduce some DSL design? Like > Catalog.getTable(tablePath).on(timeStamp), > Catalog.getTable(tablePath).current() for the most current version, and > more room for further extension like timestamp range, etc. I haven't read > all the source code yet and I'm not sure if it is possible. But a > design like this will keep the Catalog API lean and the API/DSL will be > self described and easier to use. > > Best regards, > Jing > > > On Wed, May 31, 2023 at 12:08 PM Krzysztof Chmielewski < > krzysiek.chmielew...@gmail.com> wrote: > > > Ok after second though I'm retracting my previous statement about Catalog > > changes you proposed. > > I do see a benefit for Delta connector actually with this change and see > > why this could be coupled with Catalog. > > > > Delta Connector SQL support, also ships a Delta Catalog implementation > for > > Flink. > > For Delta Catalog, table schema information is fetched from underlying > > _delta_log and not stored in metastore. For time travel we actually had a > > problem, that if we would like to timetravel back to some old version, > > where schema was slightly different, then we would have a conflict since > > Catalog would return current schema and not how it was for version X. > > > > With your change, our Delta Catalog can actually fetch schema for > version X > > and send it to DeltaTableFactory. Currency, Catalog can fetch only > current > > version. What we would also need however is version (number/timestamp) > for > > this table passed to DynamicTableFactory so we could properly set Delta > > standalone library. > > > > Regards, > > Krzysztof > > > > śr., 31 maj 2023 o 10:37 Krzysztof Chmielewski < > > krzysiek.chmielew...@gmail.com> napisał(a): > > > > > Hi, > > > happy to see such a feature. > > > Small note from my end regarding Catalog changes. > > > > > > TL;DR > > > I don't think it is necessary to delegate this feature to the catalog. > I > > > think that since "timetravel" is per job/query property, its should not > > be > > > coupled with the Catalog or table definition. In my opinion this is > > > something that DynamicTableFactory only has to know about. I would > rather > > > see this feature as it is - SQL syntax enhancement but delegate clearly > > to > > > DynamicTableFactory. > > > > > > I've implemented timetravel feature for Delta Connector [1] using > > > current Flink API. > > > Docs are pending code review, but you can find them here [2] and >
Re: [VOTE] FLIP-308: Support Time Travel
+1 (binding) Best Yun Tang From: John Roesler Sent: Tuesday, June 20, 2023 1:53 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-308: Support Time Travel Thanks for this FLIP, Feng! I've been following along, and I'm +1 (non-binding). Thanks, -John On 2023/06/19 15:50:48 Leonard Xu wrote: > +1(binding) > > Best, > Leonard > > > On Jun 19, 2023, at 8:53 PM, Jing Ge wrote: > > > > +1(binding) > > > > On Mon, Jun 19, 2023 at 1:57 PM Benchao Li wrote: > > > >> +1 (binding) > >> > >> Lincoln Lee 于2023年6月19日周一 19:40写道： > >> > >>> +1 (binding) > >>> > >>> Best, > >>> Lincoln Lee > >>> > >>> > >>> yuxia 于2023年6月19日周一 19:30写道： > >>> > >>>> +1 (binding) > >>>> Thanks Feng driving it. > >>>> > >>>> Best regards, > >>>> Yuxia > >>>> > >>>> - 原始邮件 - > >>>> 发件人: "Feng Jin" > >>>> 收件人: "dev" > >>>> 发送时间: 星期一, 2023年 6 月 19日 下午 7:22:06 > >>>> 主题: [VOTE] FLIP-308: Support Time Travel > >>>> > >>>> Hi everyone > >>>> > >>>> Thanks for all the feedback about the FLIP-308: Support Time Travel[1]. > >>>> [2] is the discussion thread. > >>>> > >>>> > >>>> I'd like to start a vote for it. The vote will be open for at least 72 > >>>> hours(excluding weekends,until June 22, 12:00AM GMT) unless there is an > >>>> objection or an insufficient number of votes. > >>>> > >>>> > >>>> [1] > >>>> > >>>> > >>> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-308%3A+Support+Time+Travel > >>>> [2]https://lists.apache.org/thread/rpozdlf7469jmc7q7vc0s08pjnmscz00 > >>>> > >>>> > >>>> Best, > >>>> Feng > >>>> > >>> > >> > >> > >> -- > >> > >> Best, > >> Benchao Li > >> > >
Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener
Hi Shammon,
I like the idea in general and it will help to analysis the job lineages no
matter FlinkSQL or Flink jar jobs in production environments.
For Qingsheng's concern, I'd like the name of JobType more than
RuntimeExecutionMode, as the latter one is not easy to understand for users.
I have one more question on the lookup-join dim tables, it seems this FLIP does
not touch them, and will them become part of the List sources()
or adding another interface?
By the way, if you want to focus on job lineage instead of data column lineage
in this FLIP, why we must introduce so many column-lineage related interface
here?
Best
Yun Tang
From: Shammon FY
Sent: Sunday, June 25, 2023 16:13
To: dev@flink.apache.org
Subject: Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener
Hi Qingsheng,
Thanks for your valuable feedback.
> 1. Is there any specific use case to expose the batch / streaming info to
listeners or meta services?
I agree with you that Flink is evolving towards batch-streaming
unification, but the lifecycle of them is different. If a job processes a
bound dataset, it will end after completing the data processing, otherwise,
it will run for a long time. In our scenario, we will regularly schedule
some Flink jobs to process bound dataset and update some job information to
the lineage information for the "batch" jobs such as scheduled timestamp,
execution duration when jobs are finished, which is different from
"streaming" jobs. Currently Flink uses `RuntimeExecutionMode` and
`existsUnboundedSource` in `StreamingGraph` and `StreamingGraphGenerator`
to determine `JobType` and disjoin jobs. We can mark `JobType` as
`PublicEvolving` or use `RuntimeExecutionMode` and a boolean flag, what do
you think of it?
> 2. it’s better to be more specific here to tell users what information
they could expect to see here, instead of just a “job configuration” as
described in JavaDoc.
Thanks and I have updated the doc in FLIP.
> 3. About the IO executor in JobStatusChangedListenerFactory.Context.
I have updated the docs for io executor in
`JobStatusChangedListenerFactory.Context`, it is a regular thread pool and
executes submitted tasks in parallel. Users can submit tasks to the
executor which ensures that the submitted task can be executed before the
job exits.
> 4. I don’t quite get the LineageRelationEntity, which is just a list of
LineageEntity.
In the initial idea, the `LineageRelationEntity` is used for `DataStream`
to set additional lineage information besides source. For example, there
are table and column lineages in SQL jobs. When we build a `DataStream` job
with table source and sink, we can add table lineage in the following
method.
```
public class DataStreamSink {
public DataStreamSink setLineageSources(LineageEntity ... sources);
}
```
But we can not set column lineage for the above sink, and for the sake of
universality, we do not want to add a method similar to `addLineageColumn
(...)` in `DataStreamSink`. So I put this information into
LineageRelationEntity so that SQL and DataStream jobs can be consistent.
But as you mentioned, this approach does indeed lead to ambiguity and
complexity. So my current idea is to add the `setLineageRelation` method in
`DataStreamSink` directly without `LineageRelationEntity`, I have updated
the FLIP and please help to review it again, thanks.
> 5. I can’t find the definition of CatalogContext in the current code base
and Flink, which appears in the TableLineageEntity.
CatalogContext is defined in FLIP-294 and I have updated the FLIP
> 6. TableSinkLineageEntity exposes ModifyType, ChangelogMode and a boolean
(the “override” is quite confusing). I’m wondering if these are necessary
for meta services, as they are actually concepts defined in the runtime
level of Flink Table / SQL.
The information in `TableSinkLineageEntity` such as `ModifyType`,
`ChangelogMode` and `override` are mainly used for verification and
display. For example, Flink currently supports `INSERT`/`DELETE` and
`UPDATE`, we only want to report and update lineage for `INSERT` jobs in
our streaming & batch ETL, and display the `override` information on the UI.
Best,
Shammon FY
On Tue, Jun 20, 2023 at 6:19 PM Qingsheng Ren wrote:
> Hi Shammon,
>
> Thanks for starting this FLIP! Data lineage is a very important topic,
> which has been missing for a long time in Flink. I have some questions
> about the FLIP.
>
> About events and listeners:
>
> 1. I’m not sure if it is necessary to expose JobType to in JobCreatedEvent.
> This is an internal class in flink-runtime, and I think the correct API
> should be RuntimeExecutionMode. Furthermore, I think the boundary of batch
> and streaming becomes much more vague as Flink is evolving towards
> batch-streaming unification, so I’m concerned about exposing JobType as a
> public API. Is there any
Re: [ANNOUNCE] Flink 1.18 Feature Freeze Extended until July 24th, 2023
Thanks for driving this release and sharing the update on the feature freeze extension. Best Yun Tang From: Jing Ge Sent: Thursday, July 6, 2023 17:13 To: yuxia Cc: dev ; re...@apache.org ; snuyan...@gmail.com ; Konstantin Knauf Subject: Re: [ANNOUNCE] Flink 1.18 Feature Freeze Extended until July 24th, 2023 Thanks for driving it and sharing the update! Best regards, Jing On Thu, Jul 6, 2023 at 9:21 AM yuxia wrote: > Thanks for the update and thanks for your efforts. > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Rui Fan" <1996fan...@gmail.com> > 收件人: "dev" , re...@apache.org > 抄送: "Jing Ge" , snuyan...@gmail.com, "Konstantin > Knauf" > 发送时间: 星期四, 2023年 7 月 06日 下午 3:06:28 > 主题: Re: [ANNOUNCE] Flink 1.18 Feature Freeze Extended until July 24th, 2023 > > Thanks for the update, and thank you for your efforts for the 1.18 release! > > Best, > Rui Fan > > On Thu, Jul 6, 2023 at 2:40 PM Qingsheng Ren wrote: > > > Hi devs, > > > > Recently we collected some feedback from developers, and in order to give > > more time for polishing some important features in 1.18, we decide to > > extend the feature freezing date to: > > > > July 24th, 2023, at 00:00 CEST(UTC+2) > > > > which gives us ~2 weeks for development from now. There will be no > > extension after Jul 24, so please arrange new features in the next > release > > if they cannot be finished before the closing date. > > > > Thanks everyone for your work in 1.18! > > > > Best regards, > > Qingsheng, Jing, Konstantin and Sergey > > >
Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website
+1 for the new look of website. And for the content of FLIP, please do not paste the "Detailed designs" under the scope of "Rejected Alternatives", you can just post the pictures in the "Proposed Changes" part. Best Yun Tang From: Mohan, Deepthi Sent: Sunday, July 16, 2023 14:10 To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website @Chesnay Thank you for your feedback. An important takeaway from the previous discussion [1] and your feedback was to keep the design and text/diagram changes separate as each change for text and diagrams likely require deeper discussion. Therefore, as a first step I am proposing only UX changes with minimal text changes for the pages mentioned in the FLIP. The feedback we received from customers cover both aesthetics and functional aspects of the website. Note that most feedback is focused only on the main Flink website [2]. 1) New customers who are considering Flink have said about the website “there is a lot going on”, “looks too complicated”, “I am not sure *why* I should use this" and similar feedback. The proposed redesign in this FLIP helps partially address this category of feedback, but we may need to make the use cases and value proposition “pop” more than we have currently proposed in the redesign. I’d like to get the community’s thoughts on this. 2) On the look and feel of the website, I’ve already shared feedback prior that I am repeating here: “like a wiki page thrown together by developers.” Customers also point out other related Apache project websites: [3] and [4] as having “modern” user design. The proposed redesign in this FLIP will help address this feedback. Modernizing the look and feel of the website will appeal to customers who are used to what they encounter on other contemporary websites. 3) New and existing Flink developers have said “I am not sure what the diagram is supposed to depict” - referencing the main diagram on [2] and have said that the website lacks useful graphics and colors. Apart from removing the diagram on the main page [2], the current FLIP does propose major changes to diagrams in the rest of website and we can discuss them separately as they become available. I’d like to keep the FLIP focused only on the website redesign. Ultimately, to Chesnay’s point in the earlier discussion in [1], I do not want to boil the ocean with all the changes at once. In this FLIP, my proposal is to first work on the UX design as that gives us a good starting point. We can use it as a framework to make iterative changes and enhancements to diagrams and the actual website content incrementally. I’ve added a few more screenshots of additional pages to the FLIP that will give you a clearer picture of the proposed changes for the main page, What is Flink [Architecture, Applications, and Operations] pages. And finally, I am not proposing any tooling changes. [1] https://lists.apache.org/thread/c3pt00cf77lrtgt242p26lgp9l2z5yc8 [2]https://flink.apache.org/ [3] https://spark.apache.org/ [4] https://kafka.apache.org/ On 7/13/23, 6:25 AM, "Chesnay Schepler" mailto:ches...@apache.org>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. On 13/07/2023 08:07, Mohan, Deepthi wrote: > However, even these developers when explicitly asked in our conversations > often comment that the website could do with a redesign Can you go into more detail as to their specific concerns? Are there functional problems with the page, or is this just a matter of "I don't like the way it looks"? What had they trouble with? Which information was missing/unnecessary/too hard to find? The FLIP states that "/we want to modernize the website so that new and existing users can easily find information to understand what Flink is, the primary use cases where Flink is useful, and clearly understand its value proposition/." From the mock-ups I don't /really/ see how these stated goals are achieved. It mostly looks like a fresh coat of paint, with a compressed nav bar (which does reduce how much information and links we throw at people at once (which isn't necessarily bad)). Can you go into more detail w.r.t. to the proposed text/presentation/diagram changes? I assume you are not proposing any tooling changes?
Re: [VOTE] Release 2.0 must-have work items
I agree that we could downgrade "Eager state declaration" to a nice-to-have feature. For the depreciation of "queryable state", can we just rename to deprecate "current implementation of queryable state"? The feature to query the internal state is actually very useful for debugging and could provide more possibility to extend FlinkSQL more like a database. Just as Yuan replied in the previous email [1], current implementation of queryable state has many problems in design. However, I don't want to make users feel that this feature cannot be done well, and maybe we can redesign this feature. As far as I know, risingwave already support queryable state with better user experience [2]. [1] https://lists.apache.org/thread/9hmwcjb3q5c24pk3qshjvybfqk62v17m [2] https://syntaxbug.com/06a3e7c554/ Best Yun Tang From: Xintong Song Sent: Friday, July 14, 2023 13:51 To: dev@flink.apache.org Subject: Re: [VOTE] Release 2.0 must-have work items Thanks for the support, Yu. We will have the guideline before removing DataSet. We are currently prioritizing works that need to be done before the 1.18 feature freeze, and will soon get back to working on the guidelines. We expect to get the guideline ready before or soon after the 1.18 release, which will definitely be before removing DataSet in 2.0. Best, Xintong On Fri, Jul 14, 2023 at 1:06 PM Yu Li wrote: > It's great to see the discussion about what we need to improve on > (completely) switching from DataSet API to DataStream API from the user > perspective. I feel that these improvements would happen faster (only) when > we seriously prepare to remove the DataSet APIs with a target release, just > like what we are doing now. And the same applies to the SinkV1 related > discussions (smile). > > I support Xintong's opinion on keeping "Remove the DataSet APIs" a > must-have item, meantime I support Yuxia's opinion that we should > explicitly let our users know how to migrate their existing DataSet API > based applications afterwards, meaning that the guideline Xintong mentioned > is a must-have (rather than best efforts) before removing the DataSet APIs. > > Best Regards, > Yu > > > On Wed, 12 Jul 2023 at 14:00, yuxia wrote: > > > Thanks Xintong for clarification. A guideline to help users migrating > from > > DataSet to DataStream will definitely be helpful. > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "Xintong Song" > > 收件人: "dev" > > 发送时间: 星期三, 2023年 7 月 12日 上午 11:40:12 > > 主题: Re: [VOTE] Release 2.0 must-have work items > > > > @Yuxia, > > > > We are aware of the issue that you mentioned. Actually, I don't think the > > DataStream API can cover everything in the DataSet API in exactly the > same > > way, because the fundamental model, concepts and primitives of the two > sets > > of APIs are completely different. Many of the DataSet APIs, especially > > those accessing the full data set at once, do not fit in the DataStream > > concepts at all. I think what's important is that users can achieve the > > same function, even if they may need to code in a different way. > > > > We have gone through all the existing DataSet APIs, and categorized them > > into 3 kinds: > > - APIs that are well supported by DataStream API as is. E.g., map, reduce > > on grouped dataset, etc. > > - APIs that can be achieved by DataStream API as is, but with a price > > (programming complexity, or computation efficiency). E.g., reduce on full > > dataset, sort partition, etc. Admittedly, there is room for improvement > on > > these. We may keep improving these for the DataStream API, or we can > > concentrate on supporting them better in the new ProcessFunction API. > > Either way, I don't think we should block the retiring of DataSet API on > > them. > > - There are also a few APIs that cannot be supported by the DataStream > API > > as is, unless users write their custom operators from the ground up. Only > > left/rightOuterJoin and combineGroup fall into this category. I think > > combinedGroup is probably not a problem, because this is more like a > > variant of reduceGroup that allows the framework to execute more > > efficiently. As for the outer joins, depending on how badly this is > needed, > > it can be supported by emitting the non-joined entries upon triggering a > > window join. > > > > We are also planning to draft a guideline to help users migrating from > > DataSet to DataStream, which should demonstrate how users can achieve > > things like sort-partition wit
Re: [VOTE] Release 2.0 must-have work items
Hi Xintong, If the current implementation of queryable state would not block the implementation of disaggregated state-backends. I prefer to not removing the implementation until we have a better solution (maybe based on the queryable snapshot) cc @Yuan. If the list of "Remove deprecated APIs" means, we must remove the code in Flink-2.0 initial release, I would vote -1 for queryable state before we get an alternative. And I will raise the concern in the Flink roadmap discussion. Best Yun Tang From: Xintong Song Sent: Monday, July 17, 2023 10:07 To: dev@flink.apache.org Subject: Re: [VOTE] Release 2.0 must-have work items @Yun, I see your point that the ability queryable states trying to provide is meaningful but the current implementation of the feature is problematic. So what's your opinion on deprecating the current queryable state? Do you think we need to wait until there is a new implementation of queryable state to remove the current one? Or maybe the current implementation is not well functional anyway and we can treat the removal of it as independent from introducing a new one? However, I don't want to make users feel that this feature cannot be done > well, and maybe we can redesign this feature. > TBH, the impression that I got from the roadmap[1] is that the queryable state is retiring and will be replaced by the state processor api. If this is not the impression we want users to have, you probably also need to raise it in the roadmap discussion [2]. Best, Xintong [1] https://flink.apache.org/roadmap [2] https://lists.apache.org/thread/szdr4ngrfcmo7zko4917393zbqhgw0v5 On Mon, Jul 17, 2023 at 9:53 AM Xintong Song wrote: > I'd propose to downgrade "Refactor the API modules" to TBD. The original > proposal was based on the condition that we are allowed to introduce > in-place API breaking changes in release 2.0. As the migration period is > introduced, and we are no longer planning to do in-place changes / > removal for DataStream (and same for APIs in `flink-core`), we need to > re-evaluate whether it's feasible to do things like moving classes to > different module / packages, turning concrete classes into interfaces on > the API classes. > > Best, > > Xintong > > > > On Mon, Jul 17, 2023 at 1:10 AM Yun Tang wrote: > >> I agree that we could downgrade "Eager state declaration" to a >> nice-to-have feature. >> >> For the depreciation of "queryable state", can we just rename to >> deprecate "current implementation of queryable state"? The feature to query >> the internal state is actually very useful for debugging and could provide >> more possibility to extend FlinkSQL more like a database. >> >> Just as Yuan replied in the previous email [1], current implementation of >> queryable state has many problems in design. However, I don't want to make >> users feel that this feature cannot be done well, and maybe we can redesign >> this feature. As far as I know, risingwave already support queryable state >> with better user experience [2]. >> >> >> [1] https://lists.apache.org/thread/9hmwcjb3q5c24pk3qshjvybfqk62v17m >> [2] https://syntaxbug.com/06a3e7c554/ >> >> Best >> Yun Tang >> >> From: Xintong Song >> Sent: Friday, July 14, 2023 13:51 >> To: dev@flink.apache.org >> Subject: Re: [VOTE] Release 2.0 must-have work items >> >> Thanks for the support, Yu. >> >> We will have the guideline before removing DataSet. We are currently >> prioritizing works that need to be done before the 1.18 feature freeze, >> and >> will soon get back to working on the guidelines. We expect to get the >> guideline ready before or soon after the 1.18 release, which will >> definitely be before removing DataSet in 2.0. >> >> Best, >> >> Xintong >> >> >> >> On Fri, Jul 14, 2023 at 1:06 PM Yu Li wrote: >> >> > It's great to see the discussion about what we need to improve on >> > (completely) switching from DataSet API to DataStream API from the user >> > perspective. I feel that these improvements would happen faster (only) >> when >> > we seriously prepare to remove the DataSet APIs with a target release, >> just >> > like what we are doing now. And the same applies to the SinkV1 related >> > discussions (smile). >> > >> > I support Xintong's opinion on keeping "Remove the DataSet APIs" a >> > must-have item, meantime I support Yuxia's opinion that we should >> > explicitly let our users know how to migrate their existing Dat
Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang
Congratulations, Yong Fang 🙂 Best Yun Tang From: Feifan Wang Sent: Wednesday, July 26, 2023 10:36 To: dev@flink.apache.org Cc: dev@flink.apache.org Subject: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang Congratulations, Yong Fang! Best, Feifan Wang | | Feifan Wang | | zoltar9...@163.com | Replied Message | From | Yuxin Tan | | Date | 07/26/2023 10:25 | | To | | | Subject | Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang | Congratulations, Yong Fang! Best, Yuxin Yanfei Lei 于2023年7月26日周三 10:18写道： Congratulations! Best regards, Yanfei weijie guo 于2023年7月26日周三 10:10写道： Congrats, Yong Fang! Best regards, Weijie Danny Cranmer 于2023年7月26日周三 03:34写道： Congrats and welcome! Danny. On Tue, 25 Jul 2023, 16:48 Matthias Pohl, wrote: Congratulations :) On Tue, Jul 25, 2023 at 5:13 PM Jing Ge wrote: Congrats, Yong Fang! Best regards, Jing On Tue, Jul 25, 2023 at 7:35 PM Yu Li wrote: Congrats, Yong! Best Regards, Yu On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin < snuyan...@gmail.com> wrote: Congratulations, Yong Fang! On Tue, Jul 25, 2023 at 7:53 AM ConradJam 于2023年7月25日周二 12:08写道： Congratulations, Yong Fang! -- Best regards, Mang Zhang 在 2023-07-25 10:30:24，"Jark Wu" 写道： Congratulations, Yong Fang! Best, Jark On Mon, 24 Jul 2023 at 22:11, Wencong Liu < liuwencle...@163.com wrote: Congratulations! Best, Wencong Liu 在 2023-07-24 11:03:30，"Paul Lam"
Re: [DISCUSS][2.0] FLIP-349: Move RocksDB statebackend classes to o.a.f.state.rocksdb package
+1 (binding) Best Yun Tang From: Yu Li Sent: Wednesday, July 26, 2023 14:10 To: dev@flink.apache.org Subject: Re: [DISCUSS][2.0] FLIP-349: Move RocksDB statebackend classes to o.a.f.state.rocksdb package +1 Best Regards, Yu On Wed, 26 Jul 2023 at 10:47, Yanfei Lei wrote: > +1 for moving all classes in the state-backend-rocksdb module under > the classes to o.a.f.state.rocksdb package. > > I have always been curious about the relationship between > o.a.f.contrib.xx and the flink-contrib module. :) > > Best, > Yanfei > > Jing Ge 于2023年7月25日周二 17:50写道： > > > > make sense. > > > > Best regards, > > Jing > > > > On Tue, Jul 25, 2023 at 4:29 PM Stefan Richter > > wrote: > > > > > > > > +1 > > > > > > > > > > > > > On 24. Jul 2023, at 12:25, Chesnay Schepler > wrote: > > > > > > > > To properly reflect the state of the rocksdb statebackend I propose > to > > > move all classes in the state-backend-rocksdb module under the classes > to > > > o.a.f.state.rocksdb package. > > > > > > > > > > > > > > > > https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-349%253A%2BMove%2BRocksDB%2Bstatebackend%2Bclasses%2Bto%2Bo.a.f.state.rocksdb%2Bpackage&source=gmail-imap&ust=169079912800&usg=AOvVaw3OiTwgsLEiTcJpNTVM-Y8y > > > > > > >
Re: [VOTE] Release 2.0 must-have work items - Round 2
+1 (non-binding), thanks @xintong for driving this work. Best Yun Tang From: Zhu Zhu Sent: Wednesday, July 26, 2023 16:35 To: dev@flink.apache.org Subject: Re: [VOTE] Release 2.0 must-have work items - Round 2 +1 (binding) Thanks, Zhu Leonard Xu 于2023年7月26日周三 15:40写道： > > Thanks @xingtong for driving the work. > > +1(binding) > > Best, > Leonard > > > On Jul 26, 2023, at 3:18 PM, Konstantin Knauf > > wrote: > > > > Hi Xingtong, > > > > yes, I am fine with the conclusion for SourceFunction. I chatted with > > Leonard a bit last night. Let's continue this vote. > > > > Thanks for the clarification, > > > > Konstantin > > > > > > > > Am Mi., 26. Juli 2023 um 04:03 Uhr schrieb Xintong Song < > > tonysong...@gmail.com>: > > > >> Hi Konstantin, > >> > >> It seems the offline discussion has already taken place [1], and part of > >> the outcome is that removal of SourceFunction would be a *nice-to-have* > >> item for release 2.0 which may not block this *must-have* vote. Do you have > >> different opinions about the conclusions in [1]? > >> > >> If there are still concerns, and the discussion around this topic needs to > >> be continued, then I'd suggest (as I mentioned in [2]) not to further block > >> this vote (i.e. the decision on other must-have items). Release 2.0 still > >> has a long way to go, and it is likely we need to review and update the > >> list every once in a while. We can update the list with another vote if > >> later we decide to add the removal of SourceFunction to the must-have list. > >> > >> WDYT? > >> > >> Best, > >> > >> Xintong > >> > >> > >> [1] https://lists.apache.org/thread/yyw52k45x2sp1jszldtdx7hc98n72w7k > >> [2] https://lists.apache.org/thread/j5d5022ky8k5t088ffm03727o5g9x9jr > >> > >> On Tue, Jul 25, 2023 at 8:49 PM Konstantin Knauf > >> wrote: > >> > >>> I assume this vote includes a decision to not removing > >>> SourceFunction/SinkFunction in Flink 2.0 (as it has been removed from the > >>> table). If this is the case, I don't think, this discussion has > >> concluded. > >>> There are multiple contributors like myself, Martijn, Alex Fedulov and > >>> Maximilian Michels, who have indicated they would be in favor of > >>> deprecating/dropping them. This Source/Sink Function discussion seems to > >> go > >>> in circles in general. I am wondering if it makes sense to have a call > >>> about this instead of repeating mailing list discussions. > >>> > >>> Am Di., 25. Juli 2023 um 13:38 Uhr schrieb Yu Li : > >>> > >>>> +1 (binding) > >>>> > >>>> Thanks for driving this, Xintong! > >>>> > >>>> Best Regards, > >>>> Yu > >>>> > >>>> > >>>> On Sun, 23 Jul 2023 at 18:28, Yuan Mei wrote: > >>>> > >>>>> +1 (binding) > >>>>> > >>>>> Thanks for driving the discussion through and for all the efforts in > >>>>> resolving the complexities :-) > >>>>> > >>>>> Best > >>>>> Yuan > >>>>> > >>>>> On Thu, Jul 20, 2023 at 5:23 PM Xintong Song > >>>>> wrote: > >>>>> > >>>>>> Hi all, > >>>>>> > >>>>>> I'd like to start another round of VOTE for the must-have work > >> items > >>>> for > >>>>>> release 2.0 [1]. The corresponding discussion thread is [2], and > >> the > >>>>>> previous voting thread is [3]. All comments from the previous > >> voting > >>>>> thread > >>>>>> have been addressed. > >>>>>> > >>>>>> Please note that once the vote is approved, any changes to the > >>>> must-have > >>>>>> items (adding / removing must-have items, changing the priority) > >>>> requires > >>>>>> another vote. Assigning contributors / reviewers, updating > >>>> descriptions / > >>>>>> progress, changes to nice-to-have items do not require another > >> vote. > >>>>>> > >>>>>> The vote will be open until at least July 25, following the > >> consensus > >>>>>> voting process. Votes of PMC members are binding. > >>>>>> > >>>>>> Best, > >>>>>> > >>>>>> Xintong > >>>>>> > >>>>>> > >>>>>> [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release > >>>>>> > >>>>>> [2] > >> https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4 > >>>>>> > >>>>>> [3] > >> https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m > >>>>>> > >>>>> > >>>> > >>> > >>> > >>> -- > >>> https://twitter.com/snntrable > >>> https://github.com/knaufk > >>> > >> > > > > > > -- > > *Konstantin Knauf* > > knauf.konstan...@gmail.com >
Re: [VOTE] FLIP-333: Redesign Apache Flink website
+1 (binding) Best Yun Tang From: liu ron Sent: Friday, August 4, 2023 10:56 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-333: Redesign Apache Flink website +1 Best, Ron Xintong Song 于2023年8月4日周五 09:35写道： > +1 (binding) > > Best, > > Xintong > > > > On Fri, Aug 4, 2023 at 3:05 AM Hong Liang wrote: > > > +1 (binding) > > > > Thanks Deepthi! > > > > > > > > On Thu, Aug 3, 2023 at 7:44 PM Danny Cranmer > > wrote: > > > > > +1 (binding) > > > > > > Thanks Deepthi > > > > > > > > > On Thu, 3 Aug 2023, 12:03 Rui Fan, <1996fan...@gmail.com> wrote: > > > > > > > +1(binding), thanks for driving this proposal, it's cool ! > > > > > > > > Best, > > > > Rui Fan > > > > > > > > On Thu, Aug 3, 2023 at 6:06 PM Jing Ge > > > wrote: > > > > > > > > > +1, thanks for driving it! > > > > > > > > > > Best regards, > > > > > Jing > > > > > > > > > > On Thu, Aug 3, 2023 at 4:49 AM Mohan, Deepthi > > > > > > > > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Thank you all for your feedback on FLIP-333. I’d like to start a > > > vote. > > > > > > > > > > > > Discussion thread: > > > > > > https://lists.apache.org/thread/z9j0rqt61ftgbkr37gzwbjg0n4fl1hsf > > > > > > FLIP: > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-333%3A+Redesign+Apache+Flink+website > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Deepthi > > > > > > > > > > > > > > > > > > > > >
Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl
Congratulation, Matthias! Best Yun Tang From: Jark Wu Sent: Friday, August 4, 2023 15:00 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl Congratulations, Matthias! Best, Jark On Fri, 4 Aug 2023 at 14:59, Weihua Hu wrote: > Congratulations, Matthias! > > Best, > Weihua > > > On Fri, Aug 4, 2023 at 2:49 PM Yuxin Tan wrote: > > > Congratulations, Matthias! > > > > Best, > > Yuxin > > > > > > Sergey Nuyanzin 于2023年8月4日周五 14:21写道： > > > > > Congratulations, Matthias! > > > Well deserved! > > > > > > On Fri, Aug 4, 2023 at 7:59 AM liu ron wrote: > > > > > > > Congrats, Matthias! > > > > > > > > Best, > > > > Ron > > > > > > > > Shammon FY 于2023年8月4日周五 13:24写道： > > > > > > > > > Congratulations, Matthias! > > > > > > > > > > On Fri, Aug 4, 2023 at 1:13 PM Samrat Deb > > > wrote: > > > > > > > > > > > Congrats, Matthias! > > > > > > > > > > > > > > > > > > On Fri, 4 Aug 2023 at 10:13 AM, Benchao Li > > > > > wrote: > > > > > > > > > > > > > Congratulations, Matthias! > > > > > > > > > > > > > > Jing Ge 于2023年8月4日周五 12:35写道： > > > > > > > > > > > > > > > Congrats! Matthias! > > > > > > > > > > > > > > > > Best regards, > > > > > > > > Jing > > > > > > > > > > > > > > > > On Fri, Aug 4, 2023 at 12:09 PM Yangze Guo < > karma...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Congrats, Matthias! > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Yangze Guo > > > > > > > > > > > > > > > > > > On Fri, Aug 4, 2023 at 11:44 AM Qingsheng Ren < > > > re...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Congratulations, Matthias! This is absolutely well > > deserved. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Qingsheng > > > > > > > > > > > > > > > > > > > > On Fri, Aug 4, 2023 at 11:31 AM Rui Fan < > > > 1996fan...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Congratulations Matthias, well deserved! > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Rui Fan > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 4, 2023 at 11:30 AM Leonard Xu < > > > > xbjt...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Congratulations, Matthias. > > > > > > > > > > > > > > > > > > > > > > > > Well deserved ^_^ > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Leonard > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Aug 4, 2023, at 11:18 AM, Xintong Song < > > > > > > > tonysong...@gmail.com > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > On behalf of the PMC, I'm very happy to announce > that > > > > > > Matthias > > > > > > > > > Pohl has > > > > > > > > > > > > > joined the Flink PMC! > > > > > > > > > > > > > > > > > > > > > > > > > > Matthias has been consistently contributing to the > > > > project > > > > > > > since > > > > > > > > > Sep > > > > > > > > > > > > 2020, > > > > > > > > > > > > > and became a committer in Dec 2021. He mainly works > > in > > > > > > Flink's > > > > > > > > > > > > distributed > > > > > > > > > > > > > coordination and high availability areas. He has > > worked > > > > on > > > > > > many > > > > > > > > > FLIPs > > > > > > > > > > > > > including FLIP195/270/285. He helped a lot with the > > > > release > > > > > > > > > management, > > > > > > > > > > > > > being one of the Flink 1.17 release managers and > also > > > > very > > > > > > > active > > > > > > > > > in > > > > > > > > > > > > Flink > > > > > > > > > > > > > 1.18 / 2.0 efforts. He also contributed a lot to > > > > improving > > > > > > the > > > > > > > > > build > > > > > > > > > > > > > stability. > > > > > > > > > > > > > > > > > > > > > > > > > > Please join me in congratulating Matthias! > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong (on behalf of the Apache Flink PMC) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Best, > > > > > > > Benchao Li > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Sergey > > > > > >
Re: [ANNOUNCE] New Apache Flink Committer - Weihua Hu
Congratulations, Weihua! Best Yun Tang From: Jark Wu Sent: Friday, August 4, 2023 15:00 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] New Apache Flink Committer - Weihua Hu Congratulations, Weihua! Best, Jark On Fri, 4 Aug 2023 at 14:48, Yuxin Tan wrote: > Congratulations Weihua! > > Best, > Yuxin > > > Junrui Lee 于2023年8月4日周五 14:28写道： > > > Congrats, Weihua! > > Best, > > Junrui > > > > Geng Biao 于2023年8月4日周五 14:25写道： > > > > > Congrats, Weihua! > > > Best, > > > Biao Geng > > > > > > 发送自 Outlook for iOS<https://aka.ms/o0ukef> > > > > > > 发件人: 周仁祥 > > > 发送时间: Friday, August 4, 2023 2:23:42 PM > > > 收件人: dev@flink.apache.org > > > 抄送: Weihua Hu > > > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Weihua Hu > > > > > > Congratulations, Weihua~ > > > > > > > 2023年8月4日 14:21，Sergey Nuyanzin 写道： > > > > > > > > Congratulations, Weihua! > > > > > > > > On Fri, Aug 4, 2023 at 8:03 AM Chen Zhanghao < > > zhanghao.c...@outlook.com> > > > > wrote: > > > > > > > >> Congratulations, Weihua! > > > >> > > > >> Best, > > > >> Zhanghao Chen > > > >> > > > >> 发件人: Xintong Song > > > >> 发送时间: 2023年8月4日 11:18 > > > >> 收件人: dev > > > >> 抄送: Weihua Hu > > > >> 主题: [ANNOUNCE] New Apache Flink Committer - Weihua Hu > > > >> > > > >> Hi everyone, > > > >> > > > >> On behalf of the PMC, I'm very happy to announce Weihua Hu as a new > > > Flink > > > >> Committer! > > > >> > > > >> Weihua has been consistently contributing to the project since May > > > 2022. He > > > >> mainly works in Flink's distributed coordination areas. He is the > main > > > >> contributor of FLIP-298 and many other improvements in large-scale > job > > > >> scheduling and improvements. He is also quite active in mailing > lists, > > > >> participating discussions and answering user questions. > > > >> > > > >> Please join me in congratulating Weihua! > > > >> > > > >> Best, > > > >> > > > >> Xintong (on behalf of the Apache Flink PMC) > > > >> > > > > > > > > > > > > -- > > > > Best regards, > > > > Sergey > > > > > > > > >
Re: [ANNOUNCE] New Apache Flink Committer - Hangxiang Yu
Congratulations, Hangxiang! Best Yun Tang From: Danny Cranmer Sent: Monday, August 7, 2023 15:11 To: dev Subject: Re: [ANNOUNCE] New Apache Flink Committer - Hangxiang Yu Congrats Hangxiang! Welcome to the team. Danny. On Mon, 7 Aug 2023, 08:04 Rui Fan, <1996fan...@gmail.com> wrote: > Congratulations Hangxiang! > > Best, > Rui > > On Mon, Aug 7, 2023 at 2:58 PM Yuan Mei wrote: > > > On behalf of the PMC, I'm happy to announce Hangxiang Yu as a new Flink > > Committer. > > > > Hangxiang has been active in the Flink community for more than 1.5 years > > and has played an important role in developing and maintaining State and > > Checkpoint related features/components, including Generic Incremental > > Checkpoints (take great efforts to make the feature prod-ready). > Hangxiang > > is also the main driver of the FLIP-263: Resolving schema compatibility. > > > > Hangxiang is passionate about the Flink community. Besides the technical > > contribution above, he is also actively promoting Flink: talks about > > Generic > > Incremental Checkpoints in Flink Forward and Meet-up. Hangxiang also > spent > > a good amount of time supporting users, participating in Jira/mailing > list > > discussions, and reviewing code. > > > > Please join me in congratulating Hangxiang for becoming a Flink > Committer! > > > > Thanks, > > Yuan Mei (on behalf of the Flink PMC) > > >
Re: [ANNOUNCE] New Apache Flink Committer - Yanfei Lei
Congratulations, Yanfei! Best Yun Tang From: Danny Cranmer Sent: Monday, August 7, 2023 15:10 To: dev Subject: Re: [ANNOUNCE] New Apache Flink Committer - Yanfei Lei Congrats Yanfei! Welcome to the team. Danny On Mon, 7 Aug 2023, 08:03 Rui Fan, <1996fan...@gmail.com> wrote: > Congratulations Yanfei! > > Best, > Rui > > On Mon, Aug 7, 2023 at 2:56 PM Yuan Mei wrote: > > > On behalf of the PMC, I'm happy to announce Yanfei Lei as a new Flink > > Committer. > > > > Yanfei has been active in the Flink community for almost two years and > has > > played an important role in developing and maintaining State and > Checkpoint > > related features/components, including RocksDB Rescaling Performance > > Improvement and Generic Incremental Checkpoints. > > > > Yanfei also helps improve community infrastructure in many ways, > including > > migrating the Flink Daily performance benchmark to the Apache Flink slack > > channel. She is the maintainer of the benchmark and has improved its > > detection stability significantly. She is also one of the major > maintainers > > of the FrocksDB Repo and released FRocksDB 6.20.3 (part of Flink 1.17 > > release). Yanfei is a very active community member, supporting users and > > participating > > in tons of discussions on the mailing lists. > > > > Please join me in congratulating Yanfei for becoming a Flink Committer! > > > > Thanks, > > Yuan Mei (on behalf of the Flink PMC) > > >
Re: [DISCUSS] How about adding OLAP to Flink Roadmap?
Thanks to the guys from ByteDance driving this topic, which could be another big story to extend Flink's ability. In general, I think this is a great idea. However, before we move forward, I think we should first answer the question: which is the target for Flink OLAP? We run Presto/Trino and SparkSQL in the production environment for OLAP SQL analysis. Since Presto runs faster than SparkSQL in many cases, especially for ad-hoc queries at medium-sized data, we would run queries on Presto first or switch to SparkSQL for large-scale queries if necessary. Presto runs as a service and emphasis on query performance without node fault tolerance. Moreover, it leverages a pipeline-like data exchange mode instead of the classic stage blocking exchange mode, which is a bit like Flink's pipeline mode vs blocking mode. Can we say we hope Flink OLAP could target Presto/Trino in medium-sized data query, and switch to Flink batch SQL for large-scale analysis query? If so, I also think the naming of Flink OLAP looks a bit strange, as Flink batch SQL shall also serve for large-scale OLAP analysis. Best Yun Tang From: Jing Ge Sent: Thursday, August 10, 2023 13:52 To: dev@flink.apache.org Subject: Re: [DISCUSS] How about adding OLAP to Flink Roadmap? Hi Shammon, Hi Xiangyu, Thanks for bringing this to our attention. I can see this is a great proposal born from real business scenarios. +1 for it. People have been keen to use one platform to cover all their data production and consumption requirements. Flink did a great job for the production, i.e. streaming and batch processing with all excellent ecosystems. This is the big advantage for Flink to go one step further and cover the consumption part. It will turn Flink into a unified compute platform like what the Ray project(the platform behind ChatGPT, if someone is not aware of it)[1] is doing and secure Flink to be one of the most interesting open source platforms for the next decade. Frankly speaking, it will be a big change. As far as I am concerned, the following should be considered(just thought about it at the first glance, there must be more). Architecture upgrade - since we will have three capabilities(I wanted to use "engines", but it might be too early to use the big word), i.e. streaming, batch, OLAP, it might make sense to upgrade the architecture while we are building the OLAP in Flink. The unified foundation or abstraction for distributed computation should be designed and implemented underneath those capabilities. In the future, new capabilities can leverage the foundation and could be developed at a very fast pace. MPP architecture - Flink session cluster is not the MMP architecture. Commonly speaking, SNA(shared nothing architecture) is the key that could implement MPP. Flink has everything to offer SNA. That is the reason why we can consider building OLAP into or on top of the Flink. And speaking of MPP, there will be a lot of things to do, e.g. the Retrieval Architecture[2], multiple level task split, dynamic retry or even split, etc. I will not expand all those topics at this early stage. OLAP queries syntax - at least some common syntax and statements need to be implemented, e.g. cube, grouping set, over partition by, you mention it. Last but not least, there will be a big effort to upgrade the runtime features to support OLAP wrt the performance and latency. Best regards, Jing [1] https://www.ray.io/ [2] https://www.tutorialsbook.com/teradata/teradata-architecture On Thu, Aug 10, 2023 at 11:39 AM Dan Zou wrote: > Thanks for bringing up this discussion, Shammon. I would like to share > some of my observations and experiences. > > Flink has almost become the de facto standard for streaming computing, and > Flink batch have been successfully applied in some companies. If Flink can > support OLAP scenarios well, a unified engine to support streaming, batch, > and OLAP will become a reality, which is very exciting. > > Based on the status quo, Flink can be used as a primary OLAP engine, > although there is still a lot of room for optimization. This means that we > do not need to carry out large-scale renovation at the beginning, but only > gradually and continuously enhance it without affecting streaming. > > Flink OLAP can largely reuse the capabilities of Flink Batch SQL and > optimizations in OLAP can also benefit Flink Batch. If we simplify job > startup overhead and increase cross-job resource reuse (Plan reuse, > Generated class reuse, Connection reuse, etc.) on this basis, Flink will > become a good OLAP engine. > > So, I am big +1 for adding OLAP to Flink Roadmap, and I am willing to > contribute to it. > > > > 2023年8月9日 15:35，xiangyu feng 写道： > > > > Thank you Shammon for initiating this discussion. As one of the Flink > OLAP > > developers in ByteDance, I would als
Re: Proposal for Implementing Keyed Watermarks in Apache Flink
Hi Tawfik, Thanks for offering such a proposal, looking forward to your research paper! You could also ask the edit permission for Flink improvement proposals to create a new proposal if you want to contribute this to the community by yourself. [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals Best Yun Tang From: yuxia Sent: Wednesday, September 6, 2023 12:31 To: dev Subject: Re: Proposal for Implementing Keyed Watermarks in Apache Flink Hi, Tawfik Yasser. Thanks for the proposal. It sounds exciting. I can't wait the research paper for more details. Best regards, Yuxia - 原始邮件 - 发件人: "David Morávek" 收件人: "dev" 发送时间: 星期二, 2023年 9 月 05日 下午 4:36:51 主题: Re: Proposal for Implementing Keyed Watermarks in Apache Flink Hi Tawfik, It's exciting to see any ongoing research that tries to push Flink forward! The get the discussion started, can you please your paper with the community? Assessing the proposal without further context is tough. Best, D. On Mon, Sep 4, 2023 at 4:42 PM Tawfek Yasser Tawfek wrote: > Dear Apache Flink Development Team, > > I hope this email finds you well. I am writing to propose an exciting new > feature for Apache Flink that has the potential to significantly enhance > its capabilities in handling unbounded streams of events, particularly in > the context of event-time windowing. > > As you may be aware, Apache Flink has been at the forefront of Big Data > Stream processing engines, leveraging windowing techniques to manage > unbounded event streams effectively. The accuracy of the results obtained > from these streams relies heavily on the ability to gather all relevant > input within a window. At the core of this process are watermarks, which > serve as unique timestamps marking the progression of events in time. > > However, our analysis has revealed a critical issue with the current > watermark generation method in Apache Flink. This method, which operates at > the input stream level, exhibits a bias towards faster sub-streams, > resulting in the unfortunate consequence of dropped events from slower > sub-streams. Our investigations showed that Apache Flink's conventional > watermark generation approach led to an alarming data loss of approximately > 33% when 50% of the keys around the median experienced delays. This loss > further escalated to over 37% when 50% of random keys were delayed. > > In response to this issue, we have authored a research paper outlining a > novel strategy named "keyed watermarks" to address data loss and > substantially enhance data processing accuracy, achieving at least 99% > accuracy in most scenarios. > > Moreover, we have conducted comprehensive comparative studies to evaluate > the effectiveness of our strategy against the conventional watermark > generation method, specifically in terms of event-time tracking accuracy. > > We believe that implementing keyed watermarks in Apache Flink can greatly > enhance its performance and reliability, making it an even more valuable > tool for organizations dealing with complex, high-throughput data > processing tasks. > > We kindly request your consideration of this proposal. We would be eager > to discuss further details, provide the full research paper, or collaborate > closely to facilitate the integration of this feature into Apache Flink. > > Thank you for your time and attention to this proposal. We look forward to > the opportunity to contribute to the continued success and evolution of > Apache Flink. > > Best Regards, > > Tawfik Yasser > Senior Teaching Assistant @ Nile University, Egypt > Email: tyas...@nu.edu.eg > LinkedIn: https://www.linkedin.com/in/tawfikyasser/ >
Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources
Thanks for creating this FLIP, Many users have demands to configure the source parallelism just as configuring the sink parallelism via DDL. Look forward for this feature. BTW, I think setting parallelism for each operator should also be valuable. And this shall work with compiled plan [1] instead of SQL's DDL. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration Best Yun Tang From: Benchao Li Sent: Thursday, September 14, 2023 19:53 To: dev@flink.apache.org Cc: dewe...@outlook.com Subject: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources Thanks Zhanghao, Dewei for preparing the FLIP, I think this is a long awaited feature, and I appreciate your effort, especially the "Other concerns" part you listed. Regarding the parallelism of transformations following the source transformation, it's indeed a problem that we initially want to solve when we introduced this feature internally. I'd like to hear more opinions on this. Personally I'm ok to leave it out of this FLIP for the time being. Chen Zhanghao 于2023年9月14日周四 14:46写道： > > Hi Devs, > > Dewei (cced) and I would like to start a discussion on FLIP-367: Support > Setting Parallelism for Table/SQL Sources [1]. > > Currently, Flink Table/SQL jobs do not expose fine-grained control of > operator parallelism to users. FLIP-146 [2] brings us support for setting > parallelism for sinks, but except for that, one can only set a default global > parallelism and all other operators share the same parallelism. However, in > many cases, setting parallelism for sources individually is preferable: > > - Many connectors have an upper bound parallelism to efficiently ingest data. > For example, the parallelism of a Kafka source is bound by the number of > partitions, any extra tasks would be idle. > - Other operators may involve intensive computation and need a larger > parallelism. > > We propose to improve the current situation by extending the current table > source API to support setting parallelism for Table/SQL sources via connector > options. > > Looking forward to your feedback. > > [1] FLIP-367: Support Setting Parallelism for Table/SQL Sources - Apache > Flink - Apache Software > Foundation<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150> > [2] FLIP-146: Improve new TableSource and TableSink interfaces - Apache Flink > - Apache Software > Foundation<https://cwiki.apache.org/confluence/display/FLINK/FLIP-146%3A+Improve+new+TableSource+and+TableSink+interfaces> > > Best, > Zhanghao Chen -- Best, Benchao Li
Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources
Hi Zhanghao, Certainly, I think we shall leave this FLIP focus on setting the source parallelism via DDL's properties. I just want to clarify that setting parallelism for individual operators is also profitable from my experience, which is slighted in your FLIP. @ConradJam BTW, compared with SQL hint, I think using `scan.parallelism` is better to align with current `sink.parallelism`. And once we introduce such option, we can also use SQL hint of dynamic table options[1] to configure the source parallelism. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/hints/#dynamic-table-options Best Yun Tang From: ConradJam Sent: Friday, September 15, 2023 22:52 To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources + 1 Thanks for the FLIP and the discussion. I would like to ask whether to use SQL Hint syntax to set this parallelism？ Martijn Visser 于2023年9月15日周五 20:52写道： > Hi everyone, > > Thanks for the FLIP and the discussion. I find it exciting. Thanks for > pushing for this. > > Best regards, > > Martijn > > On Fri, Sep 15, 2023 at 2:25 PM Chen Zhanghao > wrote: > > > Hi Jane, > > > > Thanks for the valuable suggestions. > > > > For Q1, it's indeed an issue. Some possible ideas include introducing a > > fake transformation after the source that takes the global default > > parallelism, or simply make exec nodes to take the global default > > parallelism, but both ways prevent potential chaining opportunity and I'm > > not sure if that's good to go. We'll need to give deeper thoughts in it > and > > polish our proposal. We're also more than glad to hear your inputs on it. > > > > For Q2, scan.parallelism will take high precedence, as the more specific > > config should take higher precedence. > > > > Best, > > Zhanghao Chen > > > > 发件人: Jane Chan > > 发送时间: 2023年9月15日 11:56 > > 收件人: dev@flink.apache.org > > 抄送: dewe...@outlook.com > > 主题: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL > > Sources > > > > Hi, Zhanghao, Dewei, > > > > Thanks for initiating this discussion. This feature is valuable in > > providing more flexibility for performance tuning for SQL pipelines. > > > > Here are my two cents, > > > > 1. In the FLIP, you mentioned concerns about the parallelism of the calc > > node and concluded to "leave the behavior unchanged for now." This means > > that the calc node will use the parallelism of the source operator, > > regardless of whether the source parallelism is configured or not. If I > > understand correctly, currently, except for the sink exec node (which has > > the ability to configure its own parallelism), the rest of the exec nodes > > accept its input parallelism. From the design, I didn't see the details > > about coping with input and default parallelism for the rest of the exec > > nodes. Can you elaborate more about the details? > > > > 2. Does the configuration `table.exec.resource.default-parallelism` take > > precedence over `scan.parallelism`? > > > > Best, > > Jane > > > > On Fri, Sep 15, 2023 at 10:43 AM Yun Tang wrote: > > > > > Thanks for creating this FLIP, > > > > > > Many users have demands to configure the source parallelism just as > > > configuring the sink parallelism via DDL. Look forward for this > feature. > > > > > > BTW, I think setting parallelism for each operator should also be > > > valuable. And this shall work with compiled plan [1] instead of SQL's > > DDL. > > > > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration > > > > > > Best > > > Yun Tang > > > > > > From: Benchao Li > > > Sent: Thursday, September 14, 2023 19:53 > > > To: dev@flink.apache.org > > > Cc: dewe...@outlook.com > > > Subject: Re: [DISCUSS] FLIP-367: Support Setting Parallelism for > > Table/SQL > > > Sources > > > > > > Thanks Zhanghao, Dewei for preparing the FLIP, > > > > > > I think this is a long awaited feature, and I appreciate your effort, > > > especially the "Other concerns" part you listed. > > > > > > Regarding the parallelism of transformations following the source > > > transfo
Re: [VOTE] FLIP-312: Prometheus Sink Connector
+1 (binding) Thanks for driving this, Lorenzo. Best Yun Tang From: Hong Sent: Thursday, September 21, 2023 1:22 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-312: Prometheus Sink Connector +1 (binding) Thanks Lorenzo. Hong > On 20 Sep 2023, at 17:49, Danny Cranmer wrote: > > +1 binding. > > Thanks for picking this up Lorenzo! > > Danny > > >> On Wed, 20 Sept 2023, 16:33 Jing Ge, wrote: >> >> +1(binding) Thanks! >> >> Best regards, >> Jing >> >> On Wed, Sep 20, 2023 at 3:20 PM Martijn Visser >> wrote: >> >>> +1 (binding) >>> >>> Thanks for driving this. Cheers, M >>> >>> On Mon, Sep 18, 2023 at 1:51 PM Lorenzo Nicora >> >>> wrote: >>>> >>>> Hi All, >>>> >>>> Thanks for the feedback on FLIP-312: Prometheus Sink Connector [1]. >>>> We updated the FLIP accordingly [2]. >>>> >>>> I would like to open the vote on FLIP-312. >>>> The vote will be open for at least 72 hours unless there is an >> objection >>> or >>>> insufficient votes. >>>> >>>> >>>> [1] https://lists.apache.org/thread/tm4qqfb4fxr7bc6nq5mwty1fqz8sj39x >>>> [2] >>>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector >>>> >>>> Regards >>>> Lorenzo Nicora >>> >>
Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener
+1 (binding) Best, Yun Tang From: Yangze Guo Sent: Thursday, September 28, 2023 9:20 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener +1 (binding) Best, Yangze Guo On Tue, Sep 26, 2023 at 11:05 AM Leonard Xu wrote: > > +1(binding) > > > Best, > Leonard > > > On Sep 26, 2023, at 9:59 AM, Feng Jin wrote: > > > > +1(no-binding) > > > > > > thanks for driving this proposal > > > > > > Best, > > Feng > > > > On Mon, Sep 25, 2023 at 11:19 PM Jing Ge wrote: > > > >> +1(binding) Thanks! > >> > >> Best Regards, > >> Jing > >> > >> On Sun, Sep 24, 2023 at 10:30 PM Shammon FY wrote: > >> > >>> Hi devs, > >>> > >>> Thanks for all the feedback on FLIP-314: Support Customized Job Lineage > >>> Listener [1] in thread [2]. > >>> > >>> I would like to start a vote for it. The vote will be opened for at least > >>> 72 hours unless there is an objection or insufficient votes. > >>> > >>> [1] > >>> > >>> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener > >>> [2] https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc > >>> > >>> Best, > >>> Shammon FY > >>> > >> >
Re: access to the design doc of FLINK-12477
Hi Shuyi, Thank you for your interest in the previous design. Already CC'ed Jing and Stefan, who might have such doc access. Best Yun Tang From: Shuyi Chen Sent: Thursday, September 28, 2023 13:37 To: dev Subject: access to the design doc of FLINK-12477 Hi Devs, can anyone grant access to the design doc of FLINK-12477? Thanks a lot. https://docs.google.com/document/d/1eDpsUKv2FqwZiS1Pm6gYO5eFHScBHfULKmH1-ZEWB4g Shuyi
Re: [VOTE] FLIP-367: Support Setting Parallelism for Table/SQL Sources
+1 (binding) Best Yun Tang From: Weihua Hu Sent: Monday, October 9, 2023 12:03 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-367: Support Setting Parallelism for Table/SQL Sources +1 (binding) Best, Weihua On Mon, Oct 9, 2023 at 11:47 AM Shammon FY wrote: > +1 (binding) > > > On Mon, Oct 9, 2023 at 11:12 AM Benchao Li wrote: > > > +1 (binding) > > > > Zhanghao Chen 于2023年10月9日周一 10:20写道： > > > > > > Hi All, > > > > > > Thanks for all the feedback on FLIP-367: Support Setting Parallelism > for > > Table/SQL Sources [1][2]. > > > > > > I'd like to start a vote for FLIP-367. The vote will be open until Oct > > 12th 12:00 PM GMT) unless there is an objection or insufficient votes. > > > > > > [1] > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150 > > > [2] https://lists.apache.org/thread/gtpswl42jzv0c9o3clwqskpllnw8rh87 > > > > > > Best, > > > Zhanghao Chen > > > > > > > > -- > > > > Best, > > Benchao Li > > >
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Hi Jing, I will answer current questions. > 1. will it replace the current flame graph, i.e. the current flame graph will be deprecated and removed? Although I think the new java profiler introduced in FLIP-375 is more powerful, just as Rui has replied, I don't think it could replace current flame graph totally. > 2.does it make sense to provide the performance difference between enable and disable it? The new java profiler would not introduce any performance impact after we enable it, it will only start work when we trigger the profiling. And from our experiences, the overhead of profiling is extremely light. For Rui's question: > Are all process-level flamegraphs stored at BlobStore? Are they maintained by > JobManager after sampling? Is there cleanup strategy? Or max-save-count > strategy? Yes, we use blobstore to store the process-level flamegraph-files and maintained on taskmanager side. They flamegraph-files will be cleanup automatically once reached to rest.profiling.history-size. Best Yun Tang From: Rui Fan <1996fan...@gmail.com> Sent: Tuesday, October 10, 2023 10:10 To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers Hi Jing, > 1. will it replace the current flame graph, i.e. the current flame graph will be deprecated and removed? I think the current flame graph cannot be removed. As a core contributor to the current flame graph, and I use it almost every week. I would like to clarify the difference between the current flame graph and the flame graph proposed by FLIP-375. @The current flame graph The current flame graph is the operator level or task level, when one operator is the bottleneck of current job. We can see the current flamegraph to check what the operator is doing. It includes three types: On-CPU, Off-CPU and Mixed-Type. The Mixed-Type is very useful, it can detect why operator is slow even if the operator doesn't use CPU. For example, the operator is blocked on querying hbase. It just support the task thread, it means it cannot detect the cpu usage of other threads, such as: RocksDB Flush or compaction. This's the limitation of current flamegraph. @The flame graph proposed by FLIP-375. The flamegraph proposed by FLIP-375 works on process level, such as JobManager or TaskManager, so it can monitor all threads. Such as: rocksdb background threads. When the CPU usage of one TM is high, and all tasks are not busy. The new flamegraph will be useful. Back to the question: It includes task or operator thread, why the current flamegraph is still needed? 1. The flamegraph of process level cannot easily distinguish tasks. Especially if there are multiple slots in a TM, and different subtasks of the same task running in multiple slots, their stacks are very similar. 2. The Mixed-Type of current flamegraph may not be replaced by the process-level flame graph. Please correct me if anything is wrong, thanks~ Hi Yu, > Jobmanager allows the user to download the results of the corresponding files on taskmanager with the blob service. Are all process-level flamegraphs stored at BlobStore? Are they maintained by JobManager after sampling? Is there cleanup strategy? Or max-save-count strategy? Best, Rui On Tue, Oct 10, 2023 at 1:24 AM Jing Ge wrote: > Hi Yu, Hi Yun, > > Brilliant idea! People are keen to use it. Thanks for your proposal! I was > wondering: > > 1. will it replace the current flame graph, i.e. the current flame graph > will be deprecated and removed? > 2. does it make sense to provide the performance difference between enable > and disable it? > > Best regards, > Jing > > On Mon, Oct 9, 2023 at 1:50 PM Yu Chen wrote: > > > Hi zhanghao, > > > > Yes, agree with you. We'll take Jobmanager into consideration and update > > the FLIP later! > > > > Best, > > Yu Chen > > > > Zhanghao Chen 于2023年10月9日周一 19:22写道： > > > > > Hi Yun and Yu, > > > > > > Thanks for driving this. This would definitely help users identify > > > performance bottlenecks, especially for the cases where the bottleneck > > lies > > > in the system stack (e.g. GC), and big +1 for the downloadable > flamegraph > > > to ease sharing. I'm wondering if we could add this for the job manager > > as > > > well. In the OLAP scenario and sometimes in the streaming scenario > (when > > > there're some heavy operations during execution plan generation or in > > > operator coordinators), the JM can have bottleneck as well. > > > > > > Best, > > > Zhanghao Chen > > > > > > From: Yu Chen > > > Sent: Monday, October 9, 2023 17:24 > > &g
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Hi Jing, First, developers would accept the little overhead when debugging the performance issues. Secondly, according to the async-profiler's report, it should only have less than 3% extra cost. I also verified it with a CPU-intensive ETL Flink job with this profiler, it does not show any obvious performance regression during profiling. [1] https://github.com/async-profiler/async-profiler/issues/14 Best Yun Tang From: Jing Ge Sent: Tuesday, October 10, 2023 12:05 To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers Thanks Yun for your clarification. Especially thanks Rui for your informative elaboration. Since we will have two flame graphs, I would suggest updating Flink documentation to help users understand it and know when to use which one. The content provided by Rui is already a really good starting point. Like it! Yun Theoretically yeah, I am with you. Practically, it will help get more attraction and attention, if you can provide the real (test) metrics of how "extremely light" means, e.g. engineering mindset. Otherwise, each serious user will have to evaluate the performance on her/his own before using it in production. WDYT? Best regards, Jing On Tue, Oct 10, 2023 at 5:29 AM Yun Tang wrote: > Hi Jing, > > I will answer current questions. > > > 1. will it replace the current flame graph, i.e. the current flame graph > will be deprecated and removed? > > Although I think the new java profiler introduced in FLIP-375 is more > powerful, just as Rui has replied, I don't think it could replace current > flame graph totally. > > > > 2.does it make sense to provide the performance difference between enable > and disable it? > > The new java profiler would not introduce any performance impact after we > enable it, it will only start work when we trigger the profiling. And from > our experiences, the overhead of profiling is extremely light. > > > > For Rui's question: > > > Are all process-level flamegraphs stored at BlobStore? Are they > maintained by JobManager after sampling? Is there cleanup strategy? Or > max-save-count strategy? > > Yes, we use blobstore to store the process-level flamegraph-files and > maintained on taskmanager side. They flamegraph-files will be cleanup > automatically once reached to rest.profiling.history-size. > > Best > Yun Tang > > > > > From: Rui Fan <1996fan...@gmail.com> > Sent: Tuesday, October 10, 2023 10:10 > To: dev@flink.apache.org > Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java > profiler on taskmanagers > > Hi Jing, > > > 1. will it replace the current flame graph, i.e. the current flame graph > will be deprecated and removed? > > I think the current flame graph cannot be removed. > > As a core contributor to the current flame graph, and I use it almost > every week. I would like to clarify the difference between the current > flame graph and the flame graph proposed by FLIP-375. > > @The current flame graph > > The current flame graph is the operator level or task level, when one > operator is the bottleneck of current job. We can see the current > flamegraph to check what the operator is doing. > > It includes three types: On-CPU, Off-CPU and Mixed-Type. The Mixed-Type > is very useful, it can detect why operator is slow even if the operator > doesn't use CPU. For example, the operator is blocked on querying hbase. > > It just support the task thread, it means it cannot detect the cpu usage of > other threads, such as: RocksDB Flush or compaction. This's the > limitation of current flamegraph. > > @The flame graph proposed by FLIP-375. > > The flamegraph proposed by FLIP-375 works on process level, such as > JobManager or TaskManager, so it can monitor all threads. Such as: > rocksdb background threads. > > When the CPU usage of one TM is high, and all tasks are not busy. > The new flamegraph will be useful. > > Back to the question: It includes task or operator thread, > why the current flamegraph is still needed? > > 1. The flamegraph of process level cannot easily distinguish tasks. > Especially if there are multiple slots in a TM, and different subtasks of > the > same task running in multiple slots, their stacks are very similar. > > 2. The Mixed-Type of current flamegraph may not be replaced by the > process-level flame graph. > > Please correct me if anything is wrong, thanks~ > > Hi Yu, > > > Jobmanager allows the user to download the results of the corresponding > files on taskmanager with the blob service. > > Are all process-level flameg
Re: Re: [DISCUSS] FLIP-373: Support Configuring Different State TTLs using SQL Hint
I think showing the TTL for operators is a nice-to-have feature, not a must one
in this FLIP. We can still get the information from the operator descriptions.
And I think we can continue the TTL showing work based on FLINK-33230 [1].
Last but not least, I prefer to throw exceptions if the TTL configuration is
mistakenly used as it will affect the data correctness.
[1] https://issues.apache.org/jira/browse/FLINK-33230
Best
Yun Tang
From: ConradJam
Sent: Wednesday, October 11, 2023 20:30
To: dev@flink.apache.org
Subject: Re: Re: [DISCUSS] FLIP-373: Support Configuring Different State TTLs
using SQL Hint
+1 TTL shows the state ttl for operators in Flink web ui can be know what
operator state
Zakelly Lan 于2023年10月11日周三 19:14写道：
> Hi Jane,
>
> The fine-grained TTL management is extremely useful for performance
> tuning, so +1 for the idea. I have a minor suggestion: would it be
> possible to provide a simple hint that allows the omission of the key?
> For example, something like "SELECT /+ STATE_TTL('1h')/", which would
> specify the TTL for all states in the 'SELECT' clause.
>
> And I also share the same concern as Feng. I am wondering if we could
> show the state ttl for operators in Flink UI.
>
>
> Best,
> Zakelly
>
> On Wed, Oct 11, 2023 at 1:27 PM Feng Jin wrote:
> >
> > Hi Jane,
> >
> > Thank you for providing this explanation.
> >
> > Another small question, since there is no exception thrown when the STATE
> > hint is set incorrectly,
> > should we somehow show that the TTL setting has taken effect?
> > For instance, exhibiting the TTL option within the operator's
> description?
> >
> > Best,
> > Feng
> >
> > On Tue, Oct 10, 2023 at 7:51 PM Xuyang wrote:
> >
> > > Hi, Jane.
> > >
> > >
> > > I think this syntax will be easier for users to set operator ttl. So
> big
> > > +1. I left some minor comments here.
> > >
> > >
> > > I notice that using STATE_TTL hints wrongly will not throw any
> exceptions.
> > > But it seems that in the current join hint scenario,
> > > if user uses an unknown table name as the chosen side, a validation
> > > exception will be thrown.
> > > Maybe we should distinguish which exceptions need to be thrown
> explicitly.
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Best！
> > > Xuyang
> > >
> > >
> > >
> > >
> > >
> > > At 2023-10-10 18:23:55, "Jane Chan" wrote:
> > > >Hi Feng,
> > > >
> > > >Thank you for your valuable comments. The reason for not including the
> > > >scenarios above is as follows:
> > > >
> > > >For <1>, the automatically inferred stateful operators are not easily
> > > >expressible in SQL. This issue was discussed in FLIP-292, and besides
> > > >ChangelogNormalize, SinkUpsertMateralizer also faces the same problem.
> > > >
> > > >For <2> and <3>, the challenge lies in internal implementation.
> During the
> > > >default_rewrite phase, the row_number expression in LogicalProject is
> > > >transformed into LogicalWindow by Calcite's
> > > >CoreRules#PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW. However,
> CalcRelSplitter
> > > >does not pass the hints as an input argument when creating
> LogicalWindow,
> > > >resulting in the loss of the hint at this step. To support this, we
> may
> > > >need to rewrite some optimization rules in Calcite, which could be a
> > > >follow-up work if required.
> > > >
> > > >Best,
> > > >Jane
> > > >
> > > >On Tue, Oct 10, 2023 at 1:40 AM Feng Jin
> wrote:
> > > >
> > > >> Hi Jane,
> > > >>
> > > >> Thank you for proposing this FLIP.
> > > >>
> > > >> I believe that this FLIP will greatly enhance the flexibility of
> setting
> > > >> state, and by setting different operators' TTL, it can also
> increase job
> > > >> stability, especially in regular join scenarios.
> > > >> The parameter design is very concise, big +1 for this, and it is
> also
> > > >> relatively easy to use for users.
> > > >>
> > > >>
> > > >> I have a small question: in the FLIP, it only mentions join and
> group.
> > > >> Should we also consider other scenarios?
> &g
Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan
Congratulations, Jane! Best Yun Tang From: Rui Fan <1996fan...@gmail.com> Sent: Monday, October 16, 2023 10:16 To: dev@flink.apache.org Cc: qingyue@gmail.com Subject: Re: [ANNOUNCE] New Apache Flink Committer - Jane Chan Congratulations Jane! Best, Rui On Mon, Oct 16, 2023 at 10:15 AM yu zelin wrote: > Congratulations! > > Best, > Yu Zelin > > > 2023年10月16日 09:58，Jark Wu 写道： > > > > Hi, everyone > > > > On behalf of the PMC, I'm very happy to announce Jane Chan as a new Flink > > Committer. > > > > Jane started code contribution in Jan 2021 and has been active in the > Flink > > community since. She authored more than 60 PRs and reviewed more than 40 > > PRs. Her contribution mainly revolves around Flink SQL, including Plan > > Advice (FLIP-280), operator-level state TTL (FLIP-292), and ALTER TABLE > > statements (FLINK-21634). Jane participated deeply in development > > discussions and also helped answer user question emails. Jane was also a > > core contributor of Flink Table Store (now Paimon) when the project was > in > > the early days. > > > > Please join me in congratulating Jane Chan for becoming a Flink > Committer! > > > > Best, > > Jark Wu (on behalf of the Flink PMC) > >
Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu
Congratulations, Ron! Best Yun Tang From: yu zelin Sent: Monday, October 16, 2023 10:16 To: dev@flink.apache.org Cc: ron9@gmail.com Subject: Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu Congratulations! Best, Yu Zelin > 2023年10月16日 09:56，Jark Wu 写道： > > Hi, everyone > > On behalf of the PMC, I'm very happy to announce Ron Liu as a new Flink > Committer. > > Ron has been continuously contributing to the Flink project for many years, > authored and reviewed a lot of codes. He mainly works on Flink SQL parts > and drove several important FLIPs, e.g., USING JAR (FLIP-214), Operator > Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a great > knowledge of the Batch SQL and improved a lot of batch performance in the > past several releases. He is also quite active in mailing lists, > participating in discussions and answering user questions. > > Please join me in congratulating Ron Liu for becoming a Flink Committer! > > Best, > Jark Wu (on behalf of the Flink PMC)
Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers
Thanks for the response, @Rui, actually async-profiler also supports memory allocation and wall-clock profiling, and I have updated the FILP to include these profiling options in the next developments. I think it deserves to be described as powerful. @David, in our internal version, we also support the perf_events option, and I think we can include it as an extension in this FLIP. For the request of dynamically changing the kernel parameters, we would not include this in the FLIP as it might cause permissions issues. Best Yun Tang From: David Christle Sent: Saturday, October 14, 2023 4:11 To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers In the Wiki, this FLIP is motivated by: - That the current flamegraph functionality can only see operator-level stack traces, while async-profiler provides CPU/allocation/locks information, along with deeper Java & system call stack information. - Low configurability (e.g. cannot set the sampling interval) + the usability is limited to visual inspection of the flamegraph whenever it happens to read out. The current built-in flamegraph is extremely valuable and easy to use. But as a regular user of async-profiler for Flink applications, I agree these deficiencies are worth improving. It's exciting that async-profiler might be built-in, since it will make using it much easier. I'm a little confused, though, about the scope of functionality we'll have with this FLIP, in particular the use of itimer only & perf_events support. One of the questions near the end of the Wiki is whether sampling with perf_events will be supported. The current answer seems to say that only itimer mode will be supported, as this mode does not rely on perf_events being enabled. However, given that an aim of the FLIP is to support configurability (e.g. sampling intervals), is it that much more work to support configurability of the event & the other common options, too? If the '-e' event flag is fixed to itimer only, we can't use wall clock/alloc/cpu profiling modes. The Wiki mentions async-profiler's JNI interface will be used, which has 'event_str' as an input. So, it seems like supporting different event types (or even multiple event types in one profile) is possible. Regarding perf_events, it's true that it's disabled in many environments. But it is possible to enable it for debugging purposes. In our Kubernetes workloads, this means adding SYS_PTRACE and SYS_ADMIN to the securityContext, deploying the job, and then running: sysctl kernel.kptr_restrict=0 sysctl kernel.perf_event_paranoid=1 before starting async-profiler. It would be nice if dynamically changing the kernel parameters was built-in to this FLIP, somehow, as well, to set these parameters correctly before profiling. If the environment restricts changing these, that's fine. We can simply report to the user via the UI that setting them failed, and that the choice of profiling configurations is limited without them. I also think it's OK if `itimer` is the default in the UI, as it works under the broadest conditions. But given the motivation in the Wiki is that async-profiler can see detailed system call stack info, allocation, etc., and that the async-profiler docs describe itimer mode as a "fallback" rather than the way the profiler is best used, it feels like this FLIP should support async-profiler's regular modes of operation & the other most-common configuration options. From my own experience, `cpu` (requiring perf_events) is a bit more accurate than `itimer`, and if I recall, and samples once per thread. `wall` is very useful to debug blocks on I/O or locks. Getting per-thread information is nice to drill down into specific parts of the Flink application, e.g. the flame graph lets me ignore the many other tasks running on TM & drill down into just the Source threads, when debugging a Source issue. Kind regards, David On Fri, Oct 13, 2023 at 1:45 AM Rui Fan <1996fan...@gmail.com> wrote: > One minor comment: > > In general, the generic java profiler includes memory analysis, > cpu, thread, deadlock, etc. The FLIP title is java profiler, but > the FLIP just supports flamegraph at process level. > So the `powerful java profiler` title may not be suitable. > Would you mind updating the FLIP title? > > Best, > Rui > > On Fri, Oct 13, 2023 at 4:34 PM Yu Chen wrote: > > > Hi all. > > If there are no further questions, we will start a vote on FLIP-375 next > > week. > > > > Best regards, > > Yu Chen > > > > > > Yu Chen 于2023年10月9日周一 17:24写道： > > > > > Hi all, > > > > > > Yun Tang and I are opening this thread to discuss our proposal to > > > integrate async-profiler's
Re: [VOTE] Release 1.18.0, release candidate #2
Hi Jing, I found the pre-built Flink binary release is built with JDK17[1]. Since the default target version is still 1.8 [2] and we did not mention that building with java17 is supported [3], do we really need to make the default binary release built with JDK17? [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/flink-1.18.0-bin-scala_2.12.tgz [2] https://github.com/apache/flink/blob/f978a77e2b9ade1e89dfb681d4b99fc13c72d2ed/pom.xml#L128 [3] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/flinkdev/building/#build-flink Best, Yun Tang From: Lijie Wang Sent: Tuesday, October 17, 2023 12:42 To: dev@flink.apache.org Subject: Re: [VOTE] Release 1.18.0, release candidate #2 +1 (non-binding) - Verified the signature and checksum - Built from the source code - Ran an example job on yarn cluster - Checked the website PR Best, Lijie Jing Ge 于2023年10月16日周一 18:43写道： > Hi everyone, > > Please review and vote on the release candidate #2 for the version > 1.18.0, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], and the pull request adding release note for > users [2] > * the official Apache source release and binary convenience releases to be > deployed to dist.apache.org [3], which are signed with the key with > fingerprint 96AE0E32CBE6E0753CE6 [4], > * all artifacts to be deployed to the Maven Central Repository [5], > * source code tag "release-1.17.0-rc2" [6], > * website pull request listing the new release and adding announcement blog > post [7]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Best regards, > Konstantin, Qingsheng, Sergey, and Jing > > [1] > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352885 > [2] https://github.com/apache/flink/pull/23527 > [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/ > [4] https://dist.apache.org/repos/dist/release/flink/KEYS > [5] https://repository.apache.org/content/repositories/orgapacheflink-1658 > [6] https://github.com/apache/flink/releases/tag/release-1.18.0-rc2 > [7] https://github.com/apache/flink-web/pull/680 >
Re: [VOTE] FLIP-375: Built-in cross-platform powerful java profiler
+1 (binding) Best, Yun Tang From: Zhanghao Chen Sent: Tuesday, October 17, 2023 15:57 To: dev@flink.apache.org Cc: myas...@live.com Subject: Re: [VOTE] FLIP-375: Built-in cross-platform powerful java profiler +1 (non-binding) Best, Zhanghao Chen From: Yu Chen Sent: Tuesday, October 17, 2023 15:51 To: dev@flink.apache.org Cc: myas...@live.com Subject: [VOTE] FLIP-375: Built-in cross-platform powerful java profiler Hi all, Thank you to everyone for the feedback and detailed comments on FLIP-375[1]. Based on the discussion thread [2], I think we are ready to take a vote to contribute this to Flink. I'd like to start a vote for it. The vote will be open for at least 72 hours (excluding weekends, unless there is an objection or an insufficient number of votes). Thanks, Yu Chen [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-375%3A+Built-in+cross-platform+powerful+java+profiler [2] https://lists.apache.org/thread/tp5vqgspsdko66dr6vm7cgtod9k2pct7
Re: [VOTE] Release 1.18.0, release candidate #2
Hi Chesnay, I ran the StateMachine example, and it failed with the exceptions below: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; at org.apache.flink.core.memory.DataOutputSerializer.wrapAsByteBuffer(DataOutputSerializer.java:51) ~[flink-dist-1.18.0.jar:1.18.0] at org.apache.flink.runtime.io.network.api.writer.RecordWriter.serializeRecord(RecordWriter.java:146) ~[flink-dist-1.18.0.jar:1.18.0] at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:107) ~[flink-dist-1.18.0.jar:1.18.0] at org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.emit(ChannelSelectorRecordWriter.java:55) ~[flink-dist-1.18.0.jar:1.18.0] at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:134) ~[flink-dist-1.18.0.jar:1.18.0] And then I checked MANIFEST.MF within flink-dist-1.18.0.jar to find the built Java version. >From the current findings, I think we shall cancel the RC2. Best Yun Tang From: Chesnay Schepler Sent: Tuesday, October 17, 2023 17:21 To: dev@flink.apache.org Subject: Re: [VOTE] Release 1.18.0, release candidate #2 IIRC releases MUST be built with JDK 8. @Yun Tang how did you discover this? There is supposed to be a check to enforce this, but weirdly enough it isn't rejecting my JDK 11 installation :/ Ah, we didn't set up the enforcer plugin correctly; while we do set a required JDK version by default this is interpreted as ">=", so anything above or equal 8 is currently accepted... On 17/10/2023 07:55, Yun Tang wrote: > Hi Jing, > > I found the pre-built Flink binary release is built with JDK17[1]. Since the > default target version is still 1.8 [2] and we did not mention that building > with java17 is supported [3], do we really need to make the default binary > release built with JDK17? > > > [1] > https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/flink-1.18.0-bin-scala_2.12.tgz > [2] > https://github.com/apache/flink/blob/f978a77e2b9ade1e89dfb681d4b99fc13c72d2ed/pom.xml#L128 > [3] > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/flinkdev/building/#build-flink > > Best, > Yun Tang > > From: Lijie Wang > Sent: Tuesday, October 17, 2023 12:42 > To: dev@flink.apache.org > Subject: Re: [VOTE] Release 1.18.0, release candidate #2 > > +1 (non-binding) > > - Verified the signature and checksum > - Built from the source code > - Ran an example job on yarn cluster > - Checked the website PR > > Best, > Lijie > > Jing Ge 于2023年10月16日周一 18:43写道： > >> Hi everyone, >> >> Please review and vote on the release candidate #2 for the version >> 1.18.0, as follows: >> [ ] +1, Approve the release >> [ ] -1, Do not approve the release (please provide specific comments) >> >> The complete staging area is available for your review, which includes: >> >> * JIRA release notes [1], and the pull request adding release note for >> users [2] >> * the official Apache source release and binary convenience releases to be >> deployed to dist.apache.org [3], which are signed with the key with >> fingerprint 96AE0E32CBE6E0753CE6 [4], >> * all artifacts to be deployed to the Maven Central Repository [5], >> * source code tag "release-1.17.0-rc2" [6], >> * website pull request listing the new release and adding announcement blog >> post [7]. >> >> The vote will be open for at least 72 hours. It is adopted by majority >> approval, with at least 3 PMC affirmative votes. >> >> Best regards, >> Konstantin, Qingsheng, Sergey, and Jing >> >> [1] >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352885 >> [2] https://github.com/apache/flink/pull/23527 >> [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/ >> [4] https://dist.apache.org/repos/dist/release/flink/KEYS >> [5] https://repository.apache.org/content/repositories/orgapacheflink-1658 >> [6] https://github.com/apache/flink/releases/tag/release-1.18.0-rc2 >> [7] https://github.com/apache/flink-web/pull/680 >>
Re: [VOTE] Release 1.18.0, release candidate #2
Hi Jing, Thanks for the update. I checked the new pre-built binary release, and it works fine now. However, I then checked the pre-built jar packages and found they are also built with JDK17, e.g., flink-runtime.jar [1]. I think we also need to update all artifacts to be deployed. [1] https://repository.apache.org/content/repositories/orgapacheflink-1658/org/apache/flink/flink-runtime/1.18.0/flink-runtime-1.18.0.jar Best, Yun Tang From: Matthias Pohl Sent: Tuesday, October 17, 2023 20:38 To: dev@flink.apache.org Cc: Yun Tang ; sergey.nuyan...@aiven.io Subject: Re: [VOTE] Release 1.18.0, release candidate #2 Great catch, Yun Tang. I created FLINK-33291 [1] to cover the issue of enforcing the JDK (and Maven) version in the release profile. [1] https://issues.apache.org/jira/browse/FLINK-33291 On Tue, Oct 17, 2023 at 1:28 PM Jing Ge wrote: Hi Folks, I have rebuilt Flink with Java 8 and replaced all binary artifacts at[3]. @Yun Tang mailto:myas...@live.com>> Would you please check it again? Thanks! BTW, there was a typo in the voting mail: >> * source code tag "release-1.17.0-rc2" [6] it should be: source code tag "release-1.18.0-rc2" [6] Sorry for the inconvenience caused. Special thanks for @sergey.nuyan...@aiven.io<mailto:sergey.nuyan...@aiven.io> mailto:sergey.nuyan...@aiven.io>> who helps double check the new binary files and found the typo. Best regards, Jing [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2 <https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/flink-1.18.0-bin-scala_2.12.tgz> [6] https://github.com/apache/flink/releases/tag/release-1.18.0-rc2 On Tue, Oct 17, 2023 at 12:27 PM Jing Ge mailto:j...@ververica.com>> wrote: > Thanks Chesnay for the hint! Appreciate it! > > Best regards, > Jing > > On Tue, Oct 17, 2023 at 11:21 AM Chesnay Schepler > mailto:ches...@apache.org>> > wrote: > >> IIRC releases MUST be built with JDK 8. >> >> @Yun Tang how did you discover this? >> >> There is supposed to be a check to enforce this, but weirdly enough it >> isn't rejecting my JDK 11 installation :/ >> Ah, we didn't set up the enforcer plugin correctly; while we do set a >> required JDK version by default this is interpreted as ">=", so anything >> above or equal 8 is currently accepted... >> >> On 17/10/2023 07:55, Yun Tang wrote: >> > Hi Jing, >> > >> > I found the pre-built Flink binary release is built with JDK17[1]. >> Since the default target version is still 1.8 [2] and we did not mention >> that building with java17 is supported [3], do we really need to make the >> default binary release built with JDK17? >> > >> > >> > [1] >> https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/flink-1.18.0-bin-scala_2.12.tgz >> > [2] >> https://github.com/apache/flink/blob/f978a77e2b9ade1e89dfb681d4b99fc13c72d2ed/pom.xml#L128 >> > [3] >> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/flinkdev/building/#build-flink >> > >> > Best, >> > Yun Tang >> > >> > From: Lijie Wang >> > mailto:wangdachui9...@gmail.com>> >> > Sent: Tuesday, October 17, 2023 12:42 >> > To: dev@flink.apache.org<mailto:dev@flink.apache.org> >> > mailto:dev@flink.apache.org>> >> > Subject: Re: [VOTE] Release 1.18.0, release candidate #2 >> > >> > +1 (non-binding) >> > >> > - Verified the signature and checksum >> > - Built from the source code >> > - Ran an example job on yarn cluster >> > - Checked the website PR >> > >> > Best, >> > Lijie >> > >> > Jing Ge 于2023年10月16日周一 18:43写道： >> > >> >> Hi everyone, >> >> >> >> Please review and vote on the release candidate #2 for the version >> >> 1.18.0, as follows: >> >> [ ] +1, Approve the release >> >> [ ] -1, Do not approve the release (please provide specific comments) >> >> >> >> The complete staging area is available for your review, which includes: >> >> >> >> * JIRA release notes [1], and the pull request adding release note for >> >> users [2] >> >> * the official Apache source release and binary convenience releases >> to be >> >> deployed to dist.apache.org<http://dist.apache.org> [3], which are signed >> >> with the key with >> >> fingerprint 96AE0E32CBE6E0753CE6 [4], >> >> * all artifacts to be deployed to the Maven
Re: [VOTE] Release 1.18.0, release candidate #2
Hi Jing, Thanks for the update, I checked the new maven repo, and the artifacts were built with JDK8 correctly. Best Yun Tang From: Jing Ge Sent: Tuesday, October 17, 2023 23:31 To: dev@flink.apache.org Cc: Matthias Pohl ; sergey.nuyan...@aiven.io Subject: Re: [VOTE] Release 1.18.0, release candidate #2 Hi Yun, I have created a new maven repo[5]. Would you like to check it out? Thanks for your support! Once I get your confirmation, I will start the second voting thread with the title "[VOTE][2] Release 1.18.0, release candidate #2", because the source code is not changed. All Devs: please ignore this thread. I will start a new one as I mentioned above. Best regards, Jing [5] https://repository.apache.org/content/repositories/orgapacheflink-1659 On Tue, Oct 17, 2023 at 3:53 PM Jing Ge wrote: > Hi Yun, > > Thanks for the feedback. Appreciate it! I will update them accordingly and > then ask you for double checking. Thanks in advance. > > Best regards, > Jing > > On Tue, Oct 17, 2023 at 2:42 PM Yun Tang wrote: > >> Hi Jing, >> >> Thanks for the update. I checked the new pre-built binary release, and it >> works fine now. However, I then checked the pre-built jar packages and >> found they are also built with JDK17, e.g., flink-runtime.jar [1]. >> >> I think we also need to update all artifacts to be deployed. >> >> >> [1] >> https://repository.apache.org/content/repositories/orgapacheflink-1658/org/apache/flink/flink-runtime/1.18.0/flink-runtime-1.18.0.jar >> >> Best, >> Yun Tang >> >> >> >> From: Matthias Pohl >> Sent: Tuesday, October 17, 2023 20:38 >> To: dev@flink.apache.org >> Cc: Yun Tang ; sergey.nuyan...@aiven.io < >> sergey.nuyan...@aiven.io> >> Subject: Re: [VOTE] Release 1.18.0, release candidate #2 >> >> Great catch, Yun Tang. I created FLINK-33291 [1] to cover the issue of >> enforcing the JDK (and Maven) version in the release profile. >> >> [1] https://issues.apache.org/jira/browse/FLINK-33291 >> >> On Tue, Oct 17, 2023 at 1:28 PM Jing Ge >> wrote: >> Hi Folks, >> >> I have rebuilt Flink with Java 8 and replaced all binary artifacts at[3]. >> @Yun >> Tang mailto:myas...@live.com>> Would you please check >> it again? Thanks! >> >> BTW, there was a typo in the voting mail: >> >> >> * source code tag "release-1.17.0-rc2" [6] >> it should be: >> source code tag "release-1.18.0-rc2" [6] >> >> Sorry for the inconvenience caused. >> >> Special thanks for @sergey.nuyan...@aiven.io> sergey.nuyan...@aiven.io> > sergey.nuyan...@aiven.io>> who >> helps double check the new binary files and found the typo. >> >> Best regards, >> Jing >> >> [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2 >> < >> https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/flink-1.18.0-bin-scala_2.12.tgz >> > >> >> [6] https://github.com/apache/flink/releases/tag/release-1.18.0-rc2 >> >> On Tue, Oct 17, 2023 at 12:27 PM Jing Ge > j...@ververica.com>> wrote: >> >> > Thanks Chesnay for the hint! Appreciate it! >> > >> > >> Best regards, >> > Jing >> > >> > On Tue, Oct 17, 2023 at 11:21 AM Chesnay Schepler > <mailto:ches...@apache.org>> >> > wrote: >> > >> >> IIRC releases MUST be built with JDK 8. >> >> >> >> @Yun Tang how did you discover this? >> >> >> >> There is supposed to be a check to enforce this, but weirdly enough it >> >> isn't rejecting my JDK 11 installation :/ >> >> Ah, we didn't set up the enforcer plugin correctly; while we do set a >> >> required JDK version by default this is interpreted as ">=", so >> anything >> >> above or equal 8 is currently accepted... >> >> >> >> On 17/10/2023 07:55, Yun Tang wrote: >> >> > Hi Jing, >> >> > >> >> > I found the pre-built Flink binary release is built with JDK17[1]. >> >> Since the default target version is still 1.8 [2] and we did not >> mention >> >> that building with java17 is supported [3], do we really need to make >> the >> >> default binary release built with JDK17? >> >> > >> >> > >> >> > [1] >> >> >> https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc2/flink-1.18.0-bin-scala_2.12.tgz
Re: [VOTE] Release 1.18.0, release candidate #3
+1 (non-binding) * Build from source code * Verify the pre-built jar packages were built with JDK8 * Verify FLIP-291 with a standalone cluster, and it works fine with StateMachine example. * Checked the signature * Viewed the PRs. Best Yun Tang From: Cheng Pan Sent: Thursday, October 19, 2023 14:38 To: dev@flink.apache.org Subject: RE: [VOTE] Release 1.18.0, release candidate #3 +1 (non-binding) We(the Apache Kyuubi community), verified that the Kyuubi Flink engine works well[1] with Flink 1.18.0 RC3. [1] https://github.com/apache/kyuubi/pull/5465 Thanks, Cheng Pan On 2023/10/19 00:26:24 Jing Ge wrote: > Hi everyone, > > Please review and vote on the release candidate #3 for the version > 1.18.0, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], and the pull request adding release note for > users [2] > * the official Apache source release and binary convenience releases to be > deployed to dist.apache.org [3], which are signed with the key with > fingerprint 96AE0E32CBE6E0753CE6 [4], > * all artifacts to be deployed to the Maven Central Repository [5], > * source code tag "release-1.18.0-rc3" [6], > * website pull request listing the new release and adding announcement blog > post [7]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Best regards, > Konstantin, Sergey, Qingsheng, and Jing > > [1] > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352885 > [2] https://github.com/apache/flink/pull/23527 > [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc3/ > [4] https://dist.apache.org/repos/dist/release/flink/KEYS > [5] https://repository.apache.org/content/repositories/orgapacheflink-1662 > [6] https://github.com/apache/flink/releases/tag/release-1.18.0-rc3 > [7] https://github.com/apache/flink-web/pull/680 >
Re: [ANNOUNCE] The Flink Speed Center and benchmark daily run are back online
Thanks for Zakelly's great work! Best Yun Tang From: Piotr Nowojski Sent: Thursday, October 19, 2023 17:56 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] The Flink Speed Center and benchmark daily run are back online Thank you! czw., 19 paź 2023 o 11:31 Konstantin Knauf napisał(a): > Thanks a lot for working on this! > > Am Do., 19. Okt. 2023 um 10:24 Uhr schrieb Zakelly Lan < > zakelly@gmail.com>: > > > Hi everyone, > > > > Flink benchmarks [1] generate daily performance reports in the Apache > > Flink slack channel (#flink-dev-benchmarks) to detect performance > > regression [2]. Those benchmarks previously were running on several > > machines donated and maintained by Ververica. Unfortunately, those > > machines were gone due to account issues [3] and the benchmarks daily > > run stopped since August 24th delaying the release of Flink 1.18 a > > bit. [4]. > > > > Ververica donated several new machines! After several weeks of work, I > > have successfully re-established the codespeed panel and benchmark > > daily run pipelines on them. At this time, we are pleased to announce > > that the Flink Speed Center and benchmark pipelines are back online. > > These new machines have a more formal management to ensure that > > previous accidents will not occur in the future. > > > > What's more, I successfully recovered historical data backed up by > > Yanfei Lei [5]. So with the old domain [6] redirected to the new > > machines, the old links that existed in previous records will still be > > valid. Besides the benchmarks with Java8 and Java11, I also added a > > pipeline for Java17 running daily. > > > > How to use it: > > We also registered a new domain name 'flink-speed.xyz' for the Flink > > Speed Center [7]. It is recommended to use the new domain in the > > future. Currently, the self-service method of triggering benchmarks is > > unavailable considering the lack of resources and potential > > vulnerabilities of Jenkins. Please contact one of Apache Flink PMCs to > > submit a benchmark. More info is updated on the wiki[8]. > > > > Daily Monitoring: > > The performance daily monitoring on the Apache Flink slack channel [2] > > is still unavailable as the benchmark results need more time to > > stabilize in the new environment. Once the baseline results become > > available for regression detection, I will enable the daily > > monitoring. > > > > Please feel free to reach out to me if you have any suggestions or > > questions. Thanks Ververica again for denoting machines! > > > > > > Best, > > Zakelly > > > > [1] https://github.com/apache/flink-benchmarks > > [2] https://lists.apache.org/thread/zok62sx4m50c79htfp18ymq5vmtgbgxj > > [3] https://issues.apache.org/jira/browse/FLINK-33052 > > [4] https://lists.apache.org//thread/5x28rp3zct4p603hm4zdwx6kfr101w38 > > [5] https://issues.apache.org/jira/browse/FLINK-30890 > > [6] http://codespeed.dak8s.net:8000 > > [7] http://flink-speed.xyz > > [8] > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847 > > > > > -- > https://twitter.com/snntrable > https://github.com/knaufk >
Re: Re: [DISCUSS] Remove legacy Paimon (TableStore) doc link from Flink web navigation
Hi devs, I am supporting to update the links. As the PR [1] author, I originally wanted to keep the legacy link to the Flink table store (0.3) as that was part of the history of Flink. And I plan to add the new Paimon's main branch link to tell users from Flink that this is the latest version. WDYT? [1] https://github.com/apache/flink-web/pull/665 Best Yun Tang From: junjie201...@gmail.com Sent: Friday, October 20, 2023 12:20 To: Jing Ge ; dev Cc: dev Subject: Re: Re: [DISCUSS] Remove legacy Paimon (TableStore) doc link from Flink web navigation +1 On Tue, Oct 17, 2023 at 11:13 AM Yong Fang wrote: > +1 > > On Tue, Oct 17, 2023 at 4:52 PM Leonard Xu wrote: > > > +1 > > > > > > > 2023年10月17日 下午4:50，Martijn Visser 写道： > > > > > > +1 > > > > > > On Tue, Oct 17, 2023 at 10:34 AM Jingsong Li > > wrote: > > >> > > >> Hi marton, > > >> > > >> Thanks for driving. +1 > > >> > > >> There is a PR to remove legacy Paimon > > >> https://github.com/apache/flink-web/pull/665 , but it hasn't been > > >> updated for a long time. > > >> > > >> Best, > > >> Jingsong > > >> > > >> On Tue, Oct 17, 2023 at 4:28 PM Márton Balassi > > wrote: > > >>> > > >>> Hi Flink & Paimon devs, > > >>> > > >>> The Flink webpage documentation navigation section still lists the > > outdated TableStore 0.3 and master docs as subproject docs (see > > attachment). I am all for advertising Paimon as a sister project of > Flink, > > but the current state is misleading in multiple ways. > > >>> > > >>> I would like to remove these obsolete links if the communities agree. > > >>> > > >>> Best, > > >>> Marton > > > > >
Re: Re: [DISCUSS] Remove legacy Paimon (TableStore) doc link from Flink web navigation
Hi Marton and Martijn, I have removed the link to the legacy Paimon (flink-table-store) but only left a link to the incubating-paimon doc. Please move to the PR review[1] for quick discussions. [1] https://github.com/apache/flink-web/pull/665 Best Yun Tang From: Márton Balassi Sent: Monday, October 23, 2023 17:06 To: dev@flink.apache.org ; myas...@live.com Cc: d...@paimon.apache.org Subject: Re: Re: [DISCUSS] Remove legacy Paimon (TableStore) doc link from Flink web navigation Hi all, Thanks for your responses. @Jingsong Li: Thanks for the reference to the web PR, I missed that. @Yun Tang: Thanks, I prefer simply removing the TableStore link from the documentation navigation of Flink, as it is not a subproject of Flink anymore - it is now its own project. It has had 2 of its own releases over a ~half a year period. I am all for having a proper links to Paimon, for example we could create a "Sister Projects" subsection in the About section of the flink.apache.org<http://flink.apache.org> webpage and have a paragraph of intro/links there or simply add Paimon related content to the Table connector docs [1]. We can make these 2 changes separately, but ideally they should be merged relatively close in time. Would you be open to updating your original PR [2] to simply remove the links or would you like me to do it instead? I am happy to review your change if you have a proposal of where to include Paimon. [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/overview/ [2] https://github.com/apache/flink-web/pull/665 On Sat, Oct 21, 2023 at 7:33 AM Yun Tang mailto:myas...@live.com>> wrote: Hi devs, I am supporting to update the links. As the PR [1] author, I originally wanted to keep the legacy link to the Flink table store (0.3) as that was part of the history of Flink. And I plan to add the new Paimon's main branch link to tell users from Flink that this is the latest version. WDYT? [1] https://github.com/apache/flink-web/pull/665 Best Yun Tang From: junjie201...@gmail.com<mailto:junjie201...@gmail.com> mailto:junjie201...@gmail.com>> Sent: Friday, October 20, 2023 12:20 To: Jing Ge ; dev mailto:dev@flink.apache.org>> Cc: dev mailto:d...@paimon.apache.org>> Subject: Re: Re: [DISCUSS] Remove legacy Paimon (TableStore) doc link from Flink web navigation +1 On Tue, Oct 17, 2023 at 11:13 AM Yong Fang mailto:zjur...@gmail.com>> wrote: > +1 > > On Tue, Oct 17, 2023 at 4:52 PM Leonard Xu > mailto:xbjt...@gmail.com>> wrote: > > > +1 > > > > > > > 2023年10月17日 下午4:50，Martijn Visser > > > mailto:martijnvis...@apache.org>> 写道： > > > > > > +1 > > > > > > On Tue, Oct 17, 2023 at 10:34 AM Jingsong Li > > > mailto:jingsongl...@gmail.com>> > > wrote: > > >> > > >> Hi marton, > > >> > > >> Thanks for driving. +1 > > >> > > >> There is a PR to remove legacy Paimon > > >> https://github.com/apache/flink-web/pull/665 , but it hasn't been > > >> updated for a long time. > > >> > > >> Best, > > >> Jingsong > > >> > > >> On Tue, Oct 17, 2023 at 4:28 PM Márton Balassi > > >> mailto:mbala...@apache.org>> > > wrote: > > >>> > > >>> Hi Flink & Paimon devs, > > >>> > > >>> The Flink webpage documentation navigation section still lists the > > outdated TableStore 0.3 and master docs as subproject docs (see > > attachment). I am all for advertising Paimon as a sister project of > Flink, > > but the current state is misleading in multiple ways. > > >>> > > >>> I would like to remove these obsolete links if the communities agree. > > >>> > > >>> Best, > > >>> Marton > > > > >
Re: [ANNOUNCE] Apache Flink 1.18.0 released
Thanks to everyone who participated in this release! Best Yun Tang From: Matthias Pohl Sent: Friday, October 27, 2023 17:23 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] Apache Flink 1.18.0 released Thanks to everyone who was involved and especially to the 1.18 release managers. :) On Fri, Oct 27, 2023 at 9:13 AM Yuepeng Pan wrote: > Thanks for the great work! Congratulations to everyone involved! > > > Best, > Yuepeng Pan > > At 2023-10-27 15:06:40, "ConradJam" wrote: > >Congratulations! > > > >Jingsong Li 于2023年10月27日周五 13:55写道： > > > >> Congratulations! > >> > >> Thanks Jing and other release managers and all contributors. > >> > >> Best, > >> Jingsong > >> > >> On Fri, Oct 27, 2023 at 1:52 PM Zakelly Lan > wrote: > >> > > >> > Congratulations and thank you all! > >> > > >> > > >> > Best, > >> > Zakelly > >> > > >> > On Fri, Oct 27, 2023 at 12:39 PM Jark Wu wrote: > >> > > > >> > > Congratulations and thanks release managers and everyone who has > >> > > contributed! > >> > > > >> > > Best, > >> > > Jark > >> > > > >> > > On Fri, 27 Oct 2023 at 12:25, Hang Ruan > >> wrote: > >> > > > >> > > > Congratulations! > >> > > > > >> > > > Best, > >> > > > Hang > >> > > > > >> > > > Samrat Deb 于2023年10月27日周五 11:50写道： > >> > > > > >> > > > > Congratulations on the great release > >> > > > > > >> > > > > Bests, > >> > > > > Samrat > >> > > > > > >> > > > > On Fri, 27 Oct 2023 at 7:59 AM, Yangze Guo > >> wrote: > >> > > > > > >> > > > > > Great work! Congratulations to everyone involved! > >> > > > > > > >> > > > > > Best, > >> > > > > > Yangze Guo > >> > > > > > > >> > > > > > On Fri, Oct 27, 2023 at 10:23 AM Qingsheng Ren < > re...@apache.org > >> > > >> > > > wrote: > >> > > > > > > > >> > > > > > > Congratulations and big THANK YOU to everyone helping with > this > >> > > > > release! > >> > > > > > > > >> > > > > > > Best, > >> > > > > > > Qingsheng > >> > > > > > > > >> > > > > > > On Fri, Oct 27, 2023 at 10:18 AM Benchao Li < > >> libenc...@apache.org> > >> > > > > > wrote: > >> > > > > > >> > >> > > > > > >> Great work, thanks everyone involved! > >> > > > > > >> > >> > > > > > >> Rui Fan <1996fan...@gmail.com> 于2023年10月27日周五 10:16写道： > >> > > > > > >> > > >> > > > > > >> > Thanks for the great work! > >> > > > > > >> > > >> > > > > > >> > Best, > >> > > > > > >> > Rui > >> > > > > > >> > > >> > > > > > >> > On Fri, Oct 27, 2023 at 10:03 AM Paul Lam < > >> paullin3...@gmail.com> > >> > > > > > wrote: > >> > > > > > >> > > >> > > > > > >> > > Finally! Thanks to all! > >> > > > > > >> > > > >> > > > > > >> > > Best, > >> > > > > > >> > > Paul Lam > >> > > > > > >> > > > >> > > > > > >> > > > 2023年10月27日 03:58，Alexander Fedulov < > >> > > > > alexander.fedu...@gmail.com> > >> > > > > > 写道： > >> > > > > > >> > > > > >> > > > > > >> > > > Great work, thanks everyone! > >> > > > > > >> > > > > >> > > > > > >> > > > Best, > >> > > > > > >> > > > Alexander > >> > > > > > >> > > > >
Re: Apply to become a Flink contributor
Hi, I think there is no so-called contributor permission currently, anyone can create a ticket but not everyone can take a ticket. You can ask some committer members in the JIRA ticket's comments to ask for taking that. If you don't know who could ask for help, please feel free to reply in this thread with the ticket you want to contribute. Best Yun Tang From: mzzx Sent: Thursday, November 2, 2023 20:43 To: dev@flink.apache.org Subject: Apply to become a Flink contributor Hi Guys, I want to contribute to Apache Flink. Would you please give me the permission as a contributor? My JIRA ID is dyccode.
Re: Request to release flink 1.6.3
Hi Rui, I could help to join the release management of 1.17.2, and I will also ask Yu Chen for help with some working items. Let's move forward and make it. Best Yun Tang From: Rui Fan <1996fan...@gmail.com> Sent: Tuesday, October 31, 2023 23:06 To: dev@flink.apache.org Subject: Re: Request to release flink 1.6.3 Thanks Vikas for the ask! Hi devs, Is anyone willing to pick up the release of 1.16.3 and 1.17.2 with me? If so, I can volunteer to release one of the versions. If no one picks it up for more than three days, I volunteer to release two versions. After it’s determined, the official discussion can be started. Looking forward to other committer or PMC join this release! Hi Matthias, Thank you for sorting out these 2 lists. 1.16.3 may be the final version of 1.16 series, it makes sense to sort out all left over issues. Also, some of release processes need PMC permission, would you mind helping it? Thanks~ Best, Rui On Tue, 31 Oct 2023 at 21:42, vikas patil wrote: > Hello Rui, > > Do we need more votes for this or are we good to go with the release of > 1.6.3 ? Please let me know. Thanks. > > -Vikas > > On Tue, Oct 24, 2023 at 9:27 AM Rui Fan <1996fan...@gmail.com> wrote: > > > Hi Vikas, > > > > Thanks for your feedback! > > > > Do you mean flink 1.16.3 instead of 1.6.3? > > > > The 1.16.2 and 1.17.1 were released on 2023-05-25, > > it’s been 5 months. And the flink community has fixed > > many bugs in the past 5 months. Usually, there is a > > fix(minor) version every three or four months, so I propose > > to release 1.16.3 and 1.17.2 now. > > > > If the community agrees to create this new patch release, I could > > volunteer as the release manager for one of the 1.16.3 or 1.17.2. > > > > Looking forward to feedback from the community, thank you > > > > [1] > > > > > https://flink.apache.org/2023/05/25/apache-flink-1.16.2-release-announcement/ > > [2] > > > > > https://flink.apache.org/2023/05/25/apache-flink-1.17.1-release-announcement/ > > > > Best, > > Rui > > > > On Tue, Oct 24, 2023 at 9:50 PM vikas patil > > wrote: > > > > > Hello All, > > > > > > Facing this FLINK-28185 < > > https://issues.apache.org/jira/browse/FLINK-28185 > > > > > > > issue for one of the flink jobs. We are running flink version 1.6.1 but > > it > > > looks like the backport <https://github.com/apache/flink/pull/22505> > to > > > 1.6 > > > was never released as 1.6.3. The latest that was released is 1.6.2 > > > <https://dlcdn.apache.org/flink/flink-1.16.2/>. By any chance we can > get > > > the 1.6.3 released ? > > > > > > Also we use the official flink docker <https://hub.docker.com/_/flink/ > > > > > image. Not sure if that needs to be updated as well manually. Thanks. > > > > > > -Vikas > > > > > >
[DISCUSS] Release 1.17.2
Hi all, I would like to discuss creating a new 1.17 patch release (1.17.2). The last 1.17 release is near half a year old, and since then, 79 tickets have been closed [1], of which 15 are blocker/critical [2]. Some of them are quite important, such as FLINK-32758 [3], FLINK-32296 [4], FLINK-32548 [5] and FLINK-33010[6]. In addition to this, FLINK-33149 [7] is important to bump snappy-java to 1.1.10.4. Although FLINK-33149 is unresolved, it was done in 1.17.2. I am not aware of any unresolved blockers and there are no in-progress tickets [8]. Please let me know if there are any issues you'd like to be included in this release but still not merged. If the community agrees to create this new patch release, I could volunteer as the release manager with Yu Chen. Since there will be another flink-1.16.3 release request during the same time, we will work with Rui Fan since many issues will be fixed in both releases. [1] https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC [2] https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC [3] https://issues.apache.org/jira/browse/FLINK-32758 [4] https://issues.apache.org/jira/browse/FLINK-32296 [5] https://issues.apache.org/jira/browse/FLINK-32548 [6] https://issues.apache.org/jira/browse/FLINK-33010 [7] https://issues.apache.org/jira/browse/FLINK-33149 [8] https://issues.apache.org/jira/projects/FLINK/versions/12353260 Best Yun Tang
Re: Re:Re: [DISCUSS] Release 1.17.2
Hi @casel.chen<mailto:casel_c...@126.com> It seems FLINK-33365 is more related to JDBC connector than the Flink runtime, and the discussion will focus on the release of Flink-1.17.2. Best Yun Tang From: casel.chen Sent: Tuesday, November 7, 2023 16:04 To: dev@flink.apache.org Cc: rui fan <1996fan...@gmail.com>; yuchen.e...@gmail.com Subject: Re:Re: [DISCUSS] Release 1.17.2 https://issues.apache.org/jira/browse/FLINK-33365 fixed or not in release 1.17.2? At 2023-11-07 09:47:29, "liu ron" wrote: >+1 > >Best, >Ron > >Jing Ge 于2023年11月6日周一 22:07写道： > >> +1 >> Thanks for your effort! >> >> Best regards, >> Jing >> >> On Mon, Nov 6, 2023 at 1:15 AM Konstantin Knauf wrote: >> >> > Thank you for picking it up! +1 >> > >> > Cheers, >> > >> > Konstantin >> > >> > Am Mo., 6. Nov. 2023 um 03:48 Uhr schrieb Yun Tang : >> > >> > > Hi all, >> > > >> > > I would like to discuss creating a new 1.17 patch release (1.17.2). The >> > > last 1.17 release is near half a year old, and since then, 79 tickets >> > have >> > > been closed [1], of which 15 are blocker/critical [2]. Some >> > > of them are quite important, such as FLINK-32758 [3], FLINK-32296 [4], >> > > FLINK-32548 [5] >> > > and FLINK-33010[6]. >> > > >> > > In addition to this, FLINK-33149 [7] is important to bump snappy-java >> to >> > > 1.1.10.4. >> > > Although FLINK-33149 is unresolved, it was done in 1.17.2. >> > > >> > > I am not aware of any unresolved blockers and there are no in-progress >> > > tickets [8]. Please let me know if there are any issues you'd like to >> be >> > > included in this release but still not merged. >> > > >> > > If the community agrees to create this new patch release, I could >> > > volunteer as the release manager with Yu Chen. >> > > >> > > Since there will be another flink-1.16.3 release request during the >> same >> > > time, we will work with Rui Fan since many issues will be fixed in both >> > > releases. >> > > >> > > [1] >> > > >> > >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC >> > > [2] >> > > >> > >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC >> > > [3] https://issues.apache.org/jira/browse/FLINK-32758 >> > > [4] https://issues.apache.org/jira/browse/FLINK-32296 >> > > [5] https://issues.apache.org/jira/browse/FLINK-32548 >> > > [6] https://issues.apache.org/jira/browse/FLINK-33010 >> > > [7] https://issues.apache.org/jira/browse/FLINK-33149 >> > > [8] https://issues.apache.org/jira/projects/FLINK/versions/12353260 >> > > >> > > Best >> > > Yun Tang >> > > >> > >> > >> > -- >> > https://twitter.com/snntrable >> > https://github.com/knaufk >> > >>
Re: [DISCUSS] FLIP-269: Properly Handling the Processing Timers on Job Termination
Sorry to pick up this discussion again after a long time.
@Yun Gao, could you share the PoC if possible?
Best
Yun Tang
On 2022/12/08 10:20:21 Piotr Nowojski wrote:
> Sounds good to me as well!
>
> Best,
> Piotrek
>
> czw., 8 gru 2022 o 09:53 Dawid Wysakowicz
> napisał(a):
>
> > Sounds like a good plan to me.
> > On 08/12/2022 08:58, Yun Gao wrote:
> >
> > Hi Dawid,
> >
> > Very thanks for the discussion and sorry for the delayed response
> > since I was hesitated on some points.
> >
> > But as a whole, with some more thought, first I agree with that adding
> > the trigger() / cancle() methods to some kind of timer object is not
> > necessary
> > for us to achieve the exactly-once for the operators. We could follow the
> > direction of "modifying the implementation of the operators" to achieve the
> > same target.
> >
> > But continue to think with this direction, it now looks to me it is also
> > not
> > needed to add the callback to the timer services:
> > 1. For InternalTimerService, the operators could just call
> > `InternalTimerService
> > #forEachProcessingTimer()` on finish to handle the pending timers.
> > 2. For the timers registered to the underlying ProcessingTimerService, at
> > least in
> > the currently listed scenarios, the operators itself knows what is the
> > remaining work
> > (e.g., the FileWriter knows if it has in-progress file to flush).
> >
> > Operators could handle the remaining timers in finish() method.
> >
> > Then the only interface we need to consider is that added to the
> > ProcessFunction. The
> > current interface also looks ok to me.
> >
> > If you think the above option works, I could first have a PoC that
> > demonstrate it is sufficient
> > to only modify the operator implementation to handling the remaining
> > workers properly on
> > finish(). If there are new issues I'll post here and we could have some
> > more discussion.
> >
> > Best,
> > Yun Gao
> >
> >
> > --Original Mail --
> > *Sender:*Dawid Wysakowicz
> >
> > *Send Date:*Fri Dec 2 21:21:25 2022
> > *Recipients:*Dev
> > *Subject:*Re: [DISCUSS] FLIP-269: Properly Handling the Processing Timers
> > on Job Termination
> >
> >> Ad. 1
> >>
> >> I'd start with ProcessingTimerService as that's the only public
> >> interface. It is exposed in the Sink V2 interface. In this scenario it
> >> would be the Sink interface that need to extend from a EOFTimersHandler. I
> >> believe it would be hard to pass it from there to the ProcessingTimeService
> >> as it is passed from the outside e.g. in the ProcessingTimeServiceAware.
> >> For that reason I'd go with a registration method in that interface.
> >>
> >> In ProcessFunction I'd go with a mixin approach, so a ProcessFunction can
> >> extend from EOFTimersHandler. I'd do that because ProcessFunction does not
> >> have an init/open method where we could register the handler.
> >>
> >> On operator level I'd have a registration method in InternalTimerService.
> >> I believe that's the only way to handle the above ProcessFunction aproach.
> >> E.g. in KeyedProcessOperator you need to check if the UDF extend from the
> >> interface not the operator itself.
> >>
> >> Ad. 2
> >>
> >> I'd go with
> >>
> >> *(Keyed)ProcessFunction:*
> >>
> >> interface EOFTimersHandler {
> >>
> >> void handleProcessingTimer(long timestamp, Context);
> >>
> >> }
> >>
> >> interface Context {
> >> public abstract void output(OutputTag outputTag, X value);
> >>
> >> public abstract K getCurrentKey();
> >>
> >> // we can extend it for waitFor later
> >>
> >> }
> >>
> >> *ProcessingTimeService: *
> >>
> >> interface EOFTimersHandler {
> >>
> >> void handleProcessingTimer(long timestamp, Context);
> >>
> >> }
> >>
> >> interface Context {
> >>
> >> // we can extend it for waitFor later
> >>
> >> }
> >>
> >> *InternalTimeService:*
> >>
> >> interface EOFTimersHandler {
> >>
> >> void handleProcessingTimer(InternalTimer timer Context);
> >>
> >> }
> >>
> >> interface Context {
> >>
> >> // w
Re: [DISCUSS] Release 1.17.2
Hi All, Thank you for your feedback! As there are no other concerns or objections, and currently I am not aware of any unresolved blockers. I will kick off the release process and start preparing for the RC1 version from today. Best Yun Tang From: Leonard Xu Sent: Wednesday, November 8, 2023 11:10 To: dev@flink.apache.org Cc: rui fan <1996fan...@gmail.com>; yuchen.e...@gmail.com Subject: Re: [DISCUSS] Release 1.17.2 Thanks Yun for picking this. +1 Best, Leonard > 2023年11月8日 上午9:23，Jingsong Li 写道： > > +1 thanks Yun > > 1.17.2 is really important. > > Best, > Jingsong > > On Tue, Nov 7, 2023 at 9:32 AM Danny Cranmer wrote: >> >> +1, thanks for picking this up. >> >> I am happy to help out with the bits you need PMC support for. >> >> Thanks, >> Danny >> >> On Tue, Nov 7, 2023 at 4:03 AM Yun Tang wrote: >> >>> Hi @casel.chen<mailto:casel_c...@126.com> >>> >>> It seems FLINK-33365 is more related to JDBC connector than the Flink >>> runtime, and the discussion will focus on the release of Flink-1.17.2. >>> >>> >>> Best >>> Yun Tang >>> >>> From: casel.chen >>> Sent: Tuesday, November 7, 2023 16:04 >>> To: dev@flink.apache.org >>> Cc: rui fan <1996fan...@gmail.com>; yuchen.e...@gmail.com < >>> yuchen.e...@gmail.com> >>> Subject: Re:Re: [DISCUSS] Release 1.17.2 >>> >>> >>> >>> >>> >>> >>> >>> https://issues.apache.org/jira/browse/FLINK-33365 fixed or not in release >>> 1.17.2? >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> At 2023-11-07 09:47:29, "liu ron" wrote: >>>> +1 >>>> >>>> Best, >>>> Ron >>>> >>>> Jing Ge 于2023年11月6日周一 22:07写道： >>>> >>>>> +1 >>>>> Thanks for your effort! >>>>> >>>>> Best regards, >>>>> Jing >>>>> >>>>> On Mon, Nov 6, 2023 at 1:15 AM Konstantin Knauf >>> wrote: >>>>> >>>>>> Thank you for picking it up! +1 >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Konstantin >>>>>> >>>>>> Am Mo., 6. Nov. 2023 um 03:48 Uhr schrieb Yun Tang >>> : >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I would like to discuss creating a new 1.17 patch release (1.17.2). >>> The >>>>>>> last 1.17 release is near half a year old, and since then, 79 >>> tickets >>>>>> have >>>>>>> been closed [1], of which 15 are blocker/critical [2]. Some >>>>>>> of them are quite important, such as FLINK-32758 [3], FLINK-32296 >>> [4], >>>>>>> FLINK-32548 [5] >>>>>>> and FLINK-33010[6]. >>>>>>> >>>>>>> In addition to this, FLINK-33149 [7] is important to bump >>> snappy-java >>>>> to >>>>>>> 1.1.10.4. >>>>>>> Although FLINK-33149 is unresolved, it was done in 1.17.2. >>>>>>> >>>>>>> I am not aware of any unresolved blockers and there are no >>> in-progress >>>>>>> tickets [8]. Please let me know if there are any issues you'd like >>> to >>>>> be >>>>>>> included in this release but still not merged. >>>>>>> >>>>>>> If the community agrees to create this new patch release, I could >>>>>>> volunteer as the release manager with Yu Chen. >>>>>>> >>>>>>> Since there will be another flink-1.16.3 release request during the >>>>> same >>>>>>> time, we will work with Rui Fan since many issues will be fixed in >>> both >>>>>>> releases. >>>>>>> >>>>>>> [1] >>>>>>> >>>>>> >>>>> >>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC >>>>>>> [2] >>>>>>> >>>>>> >>>>> >>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC >>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-32758 >>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-32296 >>>>>>> [5] https://issues.apache.org/jira/browse/FLINK-32548 >>>>>>> [6] https://issues.apache.org/jira/browse/FLINK-33010 >>>>>>> [7] https://issues.apache.org/jira/browse/FLINK-33149 >>>>>>> [8] https://issues.apache.org/jira/projects/FLINK/versions/12353260 >>>>>>> >>>>>>> Best >>>>>>> Yun Tang >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> https://twitter.com/snntrable >>>>>> https://github.com/knaufk >>>>>> >>>>> >>>
Re: Adding a new channel to Apache Flink slack workspace
+1 and glad to see more guys are using PyFlink. Best Yun Tang From: Jing Ge Sent: Wednesday, November 8, 2023 3:18 To: ro...@decodable.co.invalid Cc: dev@flink.apache.org Subject: Re: Adding a new channel to Apache Flink slack workspace +1 since there are so many questions wrt PyFlink. Best regards, Jing On Tue, Nov 7, 2023 at 2:23 AM Robin Moffatt wrote: > Since there have been no objections, can we go ahead and get this channel > created please? > > thanks :) > > On Thu, 26 Oct 2023 at 16:00, Alexander Fedulov < > alexander.fedu...@gmail.com> > wrote: > > > +1 > > I agree that the topic is distinct enough to justify a dedicated channel > > and this could lead to more active participation from people who work on > > it. > > > > Best, > > Alexander > > > > On Wed, 25 Oct 2023 at 20:03, Robin Moffatt wrote: > > > > > Hi, > > > > > > I'd like to propose adding a PyFlink channel to the Apache Flink slack > > > workspace. > > > > > > By creating a channel focussed on this it will help people find > previous > > > discussions as well as target new discussions and questions to the > > correct > > > place. PyFlink is a sufficiently distinct component to make a dedicated > > > channel viable and useful IMHO. > > > > > > There was a brief discussion of this on Slack already, the archive for > > > which can be found here: > > > > > > > > > https://www.linen.dev/s/apache-flink/t/16006099/re-surfacing-for-the-admins-https-apache-flink-slack-com-arc#1c7e-177a-4c37-8a34-a917883152ac > > > > > > thanks, > > > > > > Robin. > > > > > >
[VOTE] Release 1.17.2, release candidate #1
Hi everyone, Please review and vote on the release candidate #1 for the version 1.17.2, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint 2E0E1AB5D39D55E608071FB9F795C02A4D2482B3 [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag "release-1.17.2-rc1" [5], * website pull request listing the new release and adding announcement blog post [6]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353260 [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.17.2-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1669/ [5] https://github.com/apache/flink/releases/tag/release-1.17.2-rc1 [6] https://github.com/apache/flink-web/pull/696 Thanks, Release Manager
Re: Slack invite expired, please share an active invite
Hi Chinmay, You can try this link https://join.slack.com/t/apache-flink/shared_invite/zt-1ta0su2np-lCCV6xD7XeKjwQuHMOTBIA And the community will fix the expired invitation soon. Best Yun Tang From: bhatchin...@icloud.com.INVALID Sent: Monday, November 13, 2023 19:01 To: dev@flink.apache.org Subject: Slack invite expired, please share an active invite Hi, I want to join the Flink Slack community, but the invite in the documentation <https://flink.apache.org/what-is-flink/community/#slack> has expired. Can any existing member share an active invite? :) Thanks, Chinmay
Re: [VOTE] Release 1.16.3, release candidate #1
+1 (non-binding) * Verified signatures * Build from source code, and it looks good * Verified that jar packages are built with maven-3.2.5 and JDK8 * Reviewed the flink-web PR * Start a local standalone cluster and submit examples Best Yun Tang From: Rui Fan <1996fan...@gmail.com> Sent: Monday, November 13, 2023 18:20 To: dev Subject: [VOTE] Release 1.16.3, release candidate #1 Hi everyone, Please review and vote on the release candidate #1 for the version 1.16.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint B2D64016B940A7E0B9B72E0D7D0528B28037D8BC [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag "release-1.16.3-rc1" [5], * website pull request listing the new release and adding announcement blog post [6]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353259 [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.16.3-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1670/ [5] https://github.com/apache/flink/releases/tag/release-1.16.3-rc1 [6] https://github.com/apache/flink-web/pull/698 Thanks, Release Manager
Re: [DISCUSS] Towards flink-shaded release 18.0
+1, thanks Sergey for driving this work. Best Yun Tang From: Leonard Xu Sent: Thursday, November 23, 2023 9:38 To: dev Subject: Re: [DISCUSS] Towards flink-shaded release 18.0 +1, thanks Sergey for driving this. > 2023年11月23日 上午5:46，Martijn Visser 写道： > > +1. More than happy to help :) > > On Wed, Nov 22, 2023 at 9:16 PM Sergey Nuyanzin wrote: >> >> Hi everyone, >> >> I would like to start discussion about creating a new 18 release for >> flink-shaded[1]. >> >> Among others it brings fix for ZooKeeper CVE [2], >> a couple of Guava CVEs [3], [4] >> and support of jdk 21 from netty mentioned in one of >> the java 21[5] support subtasks [6] >> >> Also making a release now will allow to have enough time >> for testing before Flink 1.19 release. >> >> I would volunteer to make the release happen >> however probably I guess I will need some PMC help >> >> >> [1] https://github.com/apache/flink-shaded >> [2] https://nvd.nist.gov/vuln/detail/CVE-2023-44981 >> [3] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976 >> [4] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908 >> [5] https://issues.apache.org/jira/browse/FLINK-33163 >> [6] https://issues.apache.org/jira/browse/FLINK-1 >> >> >> >> >> >> >> -- >> Best regards, >> Sergey
Re: [DISCUSS] Contribute Flink Doris Connector to the Flink community
Hi, Di.Wu Thanks for creating this discussion. The Apache Doris community might have the most active contributors in ASF this year, and since I also contributed to doris-flink-connector before, I'm very glad to see this contribution to make the two communities work closer. I'm +1 for this proposal and please create a FLIP like [1] to kick off the discussion officially. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift Best Yun Tang From: wudi <676366...@qq.com.INVALID> Sent: Sunday, November 26, 2023 18:22 To: dev@flink.apache.org Subject: [DISCUSS] Contribute Flink Doris Connector to the Flink community Hi all, At present, Flink Connector and Flink's repository have been decoupled[1]. At the same time, the Flink-Doris-Connector[3] has been maintained based on the Apache Doris[2] community. I think the Flink Doris Connector can be migrated to the Flink community because it It is part of Flink Connectors and can also expand the ecosystem of Flink Connectors. I volunteer to move this forward if I can. [1] https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development [2] https://doris.apache.org/ [3] https://github.com/apache/doris-flink-connector -- Brs, di.wu
[RESULT] [VOTE] Release 1.17.2, release candidate #1
I'm happy to announce that we have unanimously approved this release. There are 13 approving votes, 4 of which are binding: - Rui Fan - Matthias Pohl (binding) - Danny Cranmer (binding) - Leonard Xu (binding) - Sergey Nuyanzin - Jiabao Sun - Hang Ruan - Feng Jin - Yuxin Tan - Weijie Guo - Jing Ge - Martijn Visser (binding) - Lincoln Lee There are no disapproving votes. I'll work on the steps to finalize the release and will send out the announcement as soon as that has been completed. Thanks, everyone! Best, Yun Tang
[ANNOUNCE] Apache Flink 1.17.2 released
The Apache Flink community is very happy to announce the release of Apache Flink 1.17.2, which is the second bugfix release for the Apache Flink 1.17 series. Apache Flink® Is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the improvements for this bugfix release: https://flink.apache.org/2023/11/29/apache-flink-1.17.2-release-announcement/ The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353260 We would like to thank all contributors of the Apache Flink community who made this release possible! Feel free to reach out to the release managers (or respond to this thread) with feedback on the release process. Our goal is to constantly improve the release process. Feedback on what could be improved or things that didn't go so well are appreciated. Regards, Release Manager
Re: [ANNOUNCE] Experimental Java 21 support now available on master
Thanks Sergey for the great work. I feel doubt about the conclusion that "don't try to load a savepoint from a Java 8/11/17 build due to bummping to scala-2.12.18", since the snapshotted state (operator/keyed state-backend), and most key/value serializer snapshots are generated by pure-java code. The only left part is that the developer uses scala UDF or scala types for key/value types. However, since all user-facing scala APIs have been deprecated [1], I don't think we have so many cases. Maybe we can give descriptions without such strong suggestions. Please correct me if I am wrong. [1] https://issues.apache.org/jira/browse/FLINK-29740 Best Yun Tang From: Rui Fan <1996fan...@gmail.com> Sent: Wednesday, November 29, 2023 16:43 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on master Thanks Sergey for the great work! Best, Rui On Wed, Nov 29, 2023 at 4:42 PM Leonard Xu wrote: > Cool ! > > Thanks Sergey for the great effort and all involved. > > > Best, > Leonard > > > 2023年11月29日 下午4:31，Swapnal Varma 写道： > > > > Congratulations Sergey, and everyone involved! > > > > Excited to work with and on this! > > > > Best, > > Swapnal > > > > > > On Wed, 29 Nov 2023, 13:58 Sergey Nuyanzin, wrote: > > > >> The master branch now builds and runs with Java 21 out-of-the-box. > >> > >> Notes: > >> - a nightly cron build was set up. > >> - In Java 21 builds, Scala is being bumped to 2.12.18 > >> which causes incompatibilities within Flink; > >> i.e. don't try to load a savepoint from a Java 8/11/17 build > >> - All the tests that are being skipped on Java 11/17 > >> are also skipped on Java 21. > >> > >> Huge shout-out to everyone participating > >> in review of my Java 21 related PRs > >> > >> If you run into any issues, please report it in FLINK-33163 > >> <https://issues.apache.org/jira/browse/FLINK-33163> . > >> > >> -- > >> Best regards, > >> Sergey > >> > >
Re: [ANNOUNCE] Experimental Java 21 support now available on master
Hi Sergey, You can leverage all tests extending SnapshotMigrationTestBase[1] to verify the logic. I believe all binary _metadata existing in the resources folder[2] were built by JDK8. I also create a ticket FLINK-33699[3] to track this. [1] https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SnapshotMigrationTestBase.java [2] https://github.com/apache/flink/tree/master/flink-tests/src/test/resources [3] https://issues.apache.org/jira/browse/FLINK-33699 Best Yun Tang From: Sergey Nuyanzin Sent: Wednesday, November 29, 2023 22:56 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on master thanks for the response >I feel doubt about the conclusion that "don't try to load a savepoint from a Java 8/11/17 build due to bumping to scala-2.12.18", since the snapshotted state (operator/keyed state-backend), and most key/value serializer snapshots are generated by pure-java code. >The only left part is that the developer uses scala UDF or scala types for key/value types. However, since all user-facing scala APIs have been deprecated, I don't think we have so many cases. Maybe we can give descriptions without such strong suggestions. That is the area where I feel I lack the knowledge to answer this precisely. My assumption was that statement about Java 21 regarding this should be similar to Java 17 which is almost same [1] Sorry for the inaccuracy Based on your statements I agree that the conclusion could be more relaxed. I'm curious whether there are some tests or anything which could clarify this? [1] https://lists.apache.org/thread/mz0m6wqjmqy8htob3w4469pjbg9305do On Wed, Nov 29, 2023 at 12:25 PM Yun Tang wrote: > Thanks Sergey for the great work. > > I feel doubt about the conclusion that "don't try to load a savepoint from > a Java 8/11/17 build due to bummping to scala-2.12.18", since the > snapshotted state (operator/keyed state-backend), and most key/value > serializer snapshots are generated by pure-java code. The only left part is > that the developer uses scala UDF or scala types for key/value types. > However, since all user-facing scala APIs have been deprecated [1], I don't > think we have so many cases. Maybe we can give descriptions without such > strong suggestions. > > Please correct me if I am wrong. > > > [1] https://issues.apache.org/jira/browse/FLINK-29740 > > Best > Yun Tang > > > From: Rui Fan <1996fan...@gmail.com> > Sent: Wednesday, November 29, 2023 16:43 > To: dev@flink.apache.org > Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on > master > > Thanks Sergey for the great work! > > Best, > Rui > > On Wed, Nov 29, 2023 at 4:42 PM Leonard Xu wrote: > > > Cool ! > > > > Thanks Sergey for the great effort and all involved. > > > > > > Best, > > Leonard > > > > > 2023年11月29日 下午4:31，Swapnal Varma 写道： > > > > > > Congratulations Sergey, and everyone involved! > > > > > > Excited to work with and on this! > > > > > > Best, > > > Swapnal > > > > > > > > > On Wed, 29 Nov 2023, 13:58 Sergey Nuyanzin, > wrote: > > > > > >> The master branch now builds and runs with Java 21 out-of-the-box. > > >> > > >> Notes: > > >> - a nightly cron build was set up. > > >> - In Java 21 builds, Scala is being bumped to 2.12.18 > > >> which causes incompatibilities within Flink; > > >> i.e. don't try to load a savepoint from a Java 8/11/17 build > > >> - All the tests that are being skipped on Java 11/17 > > >> are also skipped on Java 21. > > >> > > >> Huge shout-out to everyone participating > > >> in review of my Java 21 related PRs > > >> > > >> If you run into any issues, please report it in FLINK-33163 > > >> <https://issues.apache.org/jira/browse/FLINK-33163> . > > >> > > >> -- > > >> Best regards, > > >> Sergey > > >> > > > > > -- Best regards, Sergey
Re: [ANNOUNCE] Experimental Java 21 support now available on master
Hi Sergey, I checked the CI [1] which was executed with Java21, and noticed that the StatefulJobSnapshotMigrationITCase-related tests have passed, which proves what I guessed before, most checkpoints/savepoints should be restored successfully. I think we shall introduce such snapshot migration tests, which restore snapshots containing scala code. I also create a ticket focused on Java17 [2] [1] https://dev.azure.com/snuyanzin/flink/_build/results?buildId=2620&view=logs&j=0a15d512-44ac-5ba5-97ab-13a5d066c22c&t=9a028d19-6c4b-5a4e-d378-03fca149d0b1 [2] https://issues.apache.org/jira/browse/FLINK-33707 Best Yun Tang From: Sergey Nuyanzin Sent: Thursday, November 30, 2023 14:41 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on master Thanks Yun Tang One question to clarify: since the scala version was also bumped for java 17, shouldn't there be a similar task for java 17? On Thu, Nov 30, 2023 at 3:43 AM Yun Tang wrote: > Hi Sergey, > > You can leverage all tests extending SnapshotMigrationTestBase[1] to > verify the logic. I believe all binary _metadata existing in the resources > folder[2] were built by JDK8. > > I also create a ticket FLINK-33699[3] to track this. > > [1] > https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SnapshotMigrationTestBase.java > [2] > https://github.com/apache/flink/tree/master/flink-tests/src/test/resources > [3] https://issues.apache.org/jira/browse/FLINK-33699 > > Best > Yun Tang > > From: Sergey Nuyanzin > Sent: Wednesday, November 29, 2023 22:56 > To: dev@flink.apache.org > Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on > master > > thanks for the response > > > >I feel doubt about the conclusion that "don't try to load a savepoint from > a Java 8/11/17 build due to bumping to scala-2.12.18", since the > snapshotted state (operator/keyed state-backend), and most key/value > serializer snapshots are generated by pure-java code. > >The only left part is that the developer uses scala UDF or scala types for > key/value types. However, since all user-facing scala APIs have been > deprecated, I don't think we have so many cases. Maybe we can give > descriptions without such strong suggestions. > > That is the area where I feel I lack the knowledge to answer this > precisely. > My assumption was that statement about Java 21 regarding this should be > similar to Java 17 which is almost same [1] > Sorry for the inaccuracy > Based on your statements I agree that the conclusion could be more relaxed. > > I'm curious whether there are some tests or anything which could clarify > this? > > [1] https://lists.apache.org/thread/mz0m6wqjmqy8htob3w4469pjbg9305do > > On Wed, Nov 29, 2023 at 12:25 PM Yun Tang wrote: > > > Thanks Sergey for the great work. > > > > I feel doubt about the conclusion that "don't try to load a savepoint > from > > a Java 8/11/17 build due to bummping to scala-2.12.18", since the > > snapshotted state (operator/keyed state-backend), and most key/value > > serializer snapshots are generated by pure-java code. The only left part > is > > that the developer uses scala UDF or scala types for key/value types. > > However, since all user-facing scala APIs have been deprecated [1], I > don't > > think we have so many cases. Maybe we can give descriptions without such > > strong suggestions. > > > > Please correct me if I am wrong. > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-29740 > > > > Best > > Yun Tang > > > > > > From: Rui Fan <1996fan...@gmail.com> > > Sent: Wednesday, November 29, 2023 16:43 > > To: dev@flink.apache.org > > Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on > > master > > > > Thanks Sergey for the great work! > > > > Best, > > Rui > > > > On Wed, Nov 29, 2023 at 4:42 PM Leonard Xu wrote: > > > > > Cool ! > > > > > > Thanks Sergey for the great effort and all involved. > > > > > > > > > Best, > > > Leonard > > > > > > > 2023年11月29日 下午4:31，Swapnal Varma 写道： > > > > > > > > Congratulations Sergey, and everyone involved! > > > > > > > > Excited to work with and on this! > > > > > > > > Best, > > > > Swapnal > > > > > > > > > > > > On Wed, 29 Nov 2023,
Re: [DISCUSS] Release Flink 1.18.1
Thanks Jing for driving 1.18.1 release, +1 for this. Best Yun Tang From: Rui Fan <1996fan...@gmail.com> Sent: Saturday, December 9, 2023 21:46 To: dev@flink.apache.org Subject: Re: [DISCUSS] Release Flink 1.18.1 Thanks Jing for driving this release, +1 Best, Rui On Sat, Dec 9, 2023 at 7:33 AM Leonard Xu wrote: > Thanks Jing for driving this release, +1 > > Best, > Leonard > > > 2023年12月9日 上午1:23，Danny Cranmer 写道： > > > > +1 > > > > Thanks for driving this > > > > On Fri, 8 Dec 2023, 12:05 Timo Walther, wrote: > > > >> Thanks for taking care of this Jing. > >> > >> +1 to release 1.18.1 for this. > >> > >> Cheers, > >> Timo > >> > >> > >> On 08.12.23 10:00, Benchao Li wrote: > >>> I've merged FLINK-33313 to release-1.18 branch. > >>> > >>> Péter Váry 于2023年12月8日周五 16:56写道： > >>>> > >>>> Hi Jing, > >>>> Thanks for taking care of this! > >>>> +1 (non-binding) > >>>> Peter > >>>> > >>>> Sergey Nuyanzin ezt írta (időpont: 2023. dec. > >> 8., P, > >>>> 9:36): > >>>> > >>>>> Thanks Jing driving it > >>>>> +1 > >>>>> > >>>>> also +1 to include FLINK-33313 mentioned by Benchao Li > >>>>> > >>>>> On Fri, Dec 8, 2023 at 9:17 AM Benchao Li > >> wrote: > >>>>> > >>>>>> Thanks Jing for driving 1.18.1 releasing. > >>>>>> > >>>>>> I would like to include FLINK-33313[1] in 1.18.1, it's just a > bugfix, > >>>>>> not a blocker, but it's already merged into master, I plan to merge > it > >>>>>> to 1.8/1.7 branches today after the CI passes. > >>>>>> > >>>>>> [1] https://issues.apache.org/jira/browse/FLINK-33313 > >>>>>> > >>>>>> Jing Ge 于2023年12月8日周五 16:06写道： > >>>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I would like to discuss creating a new 1.18 patch release (1.18.1). > >> The > >>>>>>> last 1.18 release is nearly two months old, and since then, 37 > >> tickets > >>>>>> have > >>>>>>> been closed [1], of which 6 are blocker/critical [2]. Some of them > >> are > >>>>>>> quite important, such as FLINK-33598 [3] > >>>>>>> > >>>>>>> Most urgent and important one is FLINK-33523 [4] and according to > the > >>>>>>> discussion thread[5] on the ML, 1.18.1 should/must be released asap > >>>>> after > >>>>>>> the breaking change commit has been reverted. > >>>>>>> > >>>>>>> I am not aware of any other unresolved blockers and there are no > >>>>>> in-progress > >>>>>>> tickets [6]. > >>>>>>> Please let me know if there are any issues you'd like to be > included > >> in > >>>>>>> this release but still not merged. > >>>>>>> > >>>>>>> If the community agrees to create this new patch release, I could > >>>>>>> volunteer as the release manager. > >>>>>>> > >>>>>>> Best regards, > >>>>>>> Jing > >>>>>>> > >>>>>>> [1] > >>>>>>> > >>>>>> > >>>>> > >> > https://issues.apache.org/jira/browse/FLINK-33567?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.18.1%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC > >>>>>>> [2] > >>>>>>> > >>>>>> > >>>>> > >> > https://issues.apache.org/jira/browse/FLINK-33693?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.18.1%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC > >>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-33598 > >>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-33523 > >>>>>>> [5] > https://lists.apache.org/thread/m4c879y8mb7hbn2kkjh9h3d8g1jphh3j > >>>>>>> [6] > https://issues.apache.org/jira/projects/FLINK/versions/12353640 > >>>>>>> Thanks, > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> > >>>>>> Best, > >>>>>> Benchao Li > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Best regards, > >>>>> Sergey > >>>>> > >>> > >>> > >>> > >> > >> > >
Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an alternative for Flink's current Azure CI infrastructure
Thanks for Matthias driving this work! +1 (binding) Best Yun Tang From: Yangze Guo Sent: Tuesday, December 12, 2023 16:12 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an alternative for Flink's current Azure CI infrastructure +1 (binding) Best, Yangze Guo On Tue, Dec 12, 2023 at 3:51 PM Yuxin Tan wrote: > > +1 (non binding) > Thanks for the effort. > > Best, > Yuxin > > > Samrat Deb 于2023年12月12日周二 15:25写道： > > > +1 (non binding) > > Thanks for driving > > > > On Tue, 12 Dec 2023 at 11:59 AM, Sergey Nuyanzin > > wrote: > > > > > +1 (binding) > > > > > > Thanks for driving this > > > > > > On Tue, Dec 12, 2023, 07:22 Rui Fan <1996fan...@gmail.com> wrote: > > > > > > > +1(binding) > > > > > > > > Best, > > > > Rui > > > > > > > > On Tue, Dec 12, 2023 at 11:58 AM weijie guo > > > > > > wrote: > > > > > > > > > Thanks Matthias for this efforts. > > > > > > > > > > +1(binding) > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > Weijie > > > > > > > > > > > > > > > Matthias Pohl 于2023年12月11日周一 > > 21:51写道： > > > > > > > > > > > Hi everyone, > > > > > > I'd like to start a vote on FLIP-396 [1]. It covers enabling GitHub > > > > > Actions > > > > > > (GHA) in Apache Flink. This means that GHA workflows will run aside > > > > from > > > > > > the usual Azure CI workflows in a trial phase (which ends earliest > > > with > > > > > the > > > > > > release of Flink 1.19). Azure CI will still serve as the project's > > > > ground > > > > > > of truth until the community decides in a final vote to switch to > > GHA > > > > or > > > > > > stick to Azure CI. > > > > > > > > > > > > The related discussion thread can be found in [2]. > > > > > > > > > > > > The vote will remain open for at least 72 hours and only concluded > > if > > > > > there > > > > > > are no objections and enough (i.e. at least 3) binding votes. > > > > > > > > > > > > Matthias > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure > > > > > > [2] > > https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb > > > > > > > > > > > > -- > > > > > > > > > > > > [image: Aiven] <https://www.aiven.io> > > > > > > > > > > > > *Matthias Pohl* > > > > > > Opensource Software Engineer, *Aiven* > > > > > > matthias.p...@aiven.io| +49 170 9869525 > > > > > > aiven.io <https://www.aiven.io> | < > > > > > https://www.facebook.com/aivencloud > > > > > > > > > > > > > <https://www.linkedin.com/company/aiven/> < > > > > > > https://twitter.com/aiven_io> > > > > > > *Aiven Deutschland GmbH* > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] Release 1.18.1, release candidate #2
Thanks Jing for driving this release. +1 (non-binding) * Download artifacts and verify the signatures. * Verified the web PR * Verified the number of Python packages is 11 * Started a local cluster and verified FLIP-291 to see the rescale results. * Verified the jar packages were built with JDK8 Best Yun Tang From: Rui Fan <1996fan...@gmail.com> Sent: Thursday, December 28, 2023 10:54 To: dev@flink.apache.org Subject: Re: [VOTE] Release 1.18.1, release candidate #2 Thanks Jing for driving this release! +1(non-binding) - Downloaded artifacts - Verified signatures and sha512 - The source archives do not contain any binaries - Verified web PR - Build the source with Maven 3 and java8 (Checked the license as well) - bin/start-cluster.sh with java8, it works fine and no any unexpected LOG- Ran demo, it's fine: bin/flink runexamples/streaming/StateMachineExample.jar Best, Rui On Wed, Dec 27, 2023 at 8:45 PM Martijn Visser wrote: > Hi Jing, > > Thanks for driving this. > > +1 (binding) > > - Validated hashes > - Verified signature > - Verified that no binaries exist in the source archive > - Build the source with Maven via mvn clean install > -Pcheck-convergence -Dflink.version=1.18.1 > - Verified licenses > - Verified web PR > - Started a cluster and the Flink SQL client, successfully read and > wrote with the Kafka connector to Confluent Cloud with AVRO and Schema > Registry enabled > - Started a cluster and submitted a job that checkpoints to GCS without > problems > > Best regards, > > Martijn > > On Thu, Dec 21, 2023 at 4:55 AM gongzhongqiang > wrote: > > > > Thanks Jing Ge for driving this release. > > > > +1 (non-binding), I have checked: > > [✓] The checksums and signatures are validated > > [✓] The tag checked is fine > > [✓] Built from source is passed > > [✓] The flink-web PR is reviewed and checked > > > > > > Best， > > Zhongqiang Gong >
Re: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov
Congratulation to Alex and Happy New Year everyone! Best Yun Tang From: Rui Fan <1996fan...@gmail.com> Sent: Tuesday, January 2, 2024 21:33 To: dev@flink.apache.org Cc: Alexander Fedulov Subject: Re: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov Happy new year! Hmm, sorry for the typo in the last email. Congratulations Alex, well done! Best, Rui On Tue, 2 Jan 2024 at 20:23, Rui Fan <1996fan...@gmail.com> wrote: > Configurations Alexander! > > Best, > Rui > > On Tue, Jan 2, 2024 at 8:15 PM Maximilian Michels wrote: > >> Happy New Year everyone, >> >> I'd like to start the year off by announcing Alexander Fedulov as a >> new Flink committer. >> >> Alex has been active in the Flink community since 2019. He has >> contributed more than 100 commits to Flink, its Kubernetes operator, >> and various connectors [1][2]. >> >> Especially noteworthy are his contributions on deprecating and >> migrating the old Source API functions and test harnesses, the >> enhancement to flame graphs, the dynamic rescale time computation in >> Flink Autoscaling, as well as all the small enhancements Alex has >> contributed which make a huge difference. >> >> Beyond code contributions, Alex has been an active community member >> with his activity on the mailing lists [3][4], as well as various >> talks and blog posts about Apache Flink [5][6]. >> >> Congratulations Alex! The Flink community is proud to have you. >> >> Best, >> The Flink PMC >> >> [1] >> https://github.com/search?type=commits&q=author%3Aafedulov+org%3Aapache >> [2] >> https://issues.apache.org/jira/browse/FLINK-28229?jql=status%20in%20(Resolved%2C%20Closed)%20AND%20assignee%20in%20(afedulov)%20ORDER%20BY%20resolved%20DESC%2C%20created%20DESC >> [3] https://lists.apache.org/list?dev@flink.apache.org:lte=100M:Fedulov >> [4] https://lists.apache.org/list?u...@flink.apache.org:lte=100M:Fedulov >> [5] >> https://flink.apache.org/2020/01/15/advanced-flink-application-patterns-vol.1-case-study-of-a-fraud-detection-system/ >> [6] >> https://www.ververica.com/blog/presenting-our-streaming-concepts-introduction-to-flink-video-series >> >
Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration
Hi Zakelly, Thanks for driving this topic. I have two concerns here: 1. We shall not describe the configuration with its implementation for 'execution.checkpointing.local-copy.*' options, for hashmap state-backend, it would write two streams and for Rocksdb state-backend, it would use hard-link for backup. Thus, I think 'execution.checkpointing.local-backup.*' looks better. 2. What does the 'execution.checkpointing.data-inline-threshold' mean? It seems not so easy to understand. Best Yun Tang From: Piotr Nowojski Sent: Thursday, January 4, 2024 22:37 To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration Hi, Thanks for trying to clean this up! I don't have strong opinions on the topics discussed here, so generally speaking +1 from my side! Best, Piotrek śr., 3 sty 2024 o 04:16 Rui Fan <1996fan...@gmail.com> napisał(a): > Thanks for the feedback! > > Using the `execution.checkpointing.incremental.enabled`, > and enabling it by default sounds good to me. > > Best, > Rui > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan wrote: > > > Hi Rui, > > > > Thanks for your comments! > > > > IMO, given that the state backend can be plugably loaded (as you can > > specify a state backend factory), I prefer not providing state backend > > specified options in the framework. > > > > Secondly, the incremental checkpoint is actually a sharing file strategy > > across checkpoints, which means the state backend *could* reuse files > from > > previous cp but not *must* do so. When the state backend could not reuse > > the files, it is reasonable to fallback to a full checkpoint. > > > > Thus, I suggest we make it `execution.checkpointing.incremental` and > enable > > it by default. For those state backends not supporting this, they perform > > full checkpoints and print a warning to inform users. Users do not need > to > > pay special attention to different options to control this across > different > > state backends. This is more user-friendly in my opinion. WDYT? > > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > Hi Zakelly, > > > > > > I'm not sure whether we could add the state backend type in the > > > new key name of state.backend.incremental. It means we use > > > `execution.checkpointing.rocksdb-incremental` or > > > `execution.checkpointing.rocksdb-incremental.enabled`. > > > > > > So far, state.backend.incremental only works for rocksdb state backend. > > > And this feature or optimization is very valuable and huge for large > > > state flink jobs. I believe it's enabled for most production flink jobs > > > with large rocksdb state. > > > > > > If this option isn't generic for all state backend types, I guess we > > > can enable `execution.checkpointing.rocksdb-incremental.enabled` > > > by default in Flink 2.0. > > > > > > But if it works for all state backends, it's hard to enable it by > > default. > > > Enabling great and valuable features or improvements are useful > > > for users, especially a lot of new flink users. Out-of-the-box options > > > are good for users. > > > > > > WDYT? > > > > > > Best, > > > Rui > > > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan > > wrote: > > > > > > > Hi everyone, > > > > > > > > Thanks all for your comments! > > > > > > > > As many of you have questions about the names for boolean options, I > > > > suggest we make a naming rule for them. For now I could think of > three > > > > options: > > > > > > > > Option 1: Use enumeration options if possible. But this may cause > some > > > name > > > > collisions or confusion as we discussed and we should unify the > > statement > > > > everywhere. > > > > Option 2: Use boolean options and add 'enabled' as the suffix. > > > > Option 3: Use boolean options and ONLY add 'enabled' when there are > > more > > > > detailed configurations under the same prefix, to prevent one name > from > > > > serving as a prefix to another. > > > > > > > > I am slightly inclined to Option 3, since it is more in line with > > current > > > > practice and friendly for existing users. Also It reduces the length > of > > > &
Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
Congratulations and well deserved! Best Yun Tang From: Canbin Zheng Sent: Monday, February 24, 2020 16:07 To: dev Subject: Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer Congratulations !! Dawid Wysakowicz 于2020年2月24日周一 下午3:55写道： > Congratulations Jingsong! > > Best, > > Dawid > > On 24/02/2020 08:12, zhenya Sun wrote: > > Congratulations！！！ > > | | > > zhenya Sun > > | > > | > > toke...@126.com > > | > > 签名由网易邮箱大师定制 > > > > > > On 02/24/2020 14:35，Yu Li wrote： > > Congratulations Jingsong! Well deserved. > > > > Best Regards, > > Yu > > > > > > On Mon, 24 Feb 2020 at 14:10, Congxian Qiu > wrote: > > > > Congratulations Jingsong! > > > > Best, > > Congxian > > > > > > jincheng sun 于2020年2月24日周一 下午1:38写道： > > > > Congratulations Jingsong! > > > > Best, > > Jincheng > > > > > > Zhu Zhu 于2020年2月24日周一 上午11:55写道： > > > > Congratulations Jingsong! > > > > Thanks, > > Zhu Zhu > > > > Fabian Hueske 于2020年2月22日周六 上午1:30写道： > > > > Congrats Jingsong! > > > > Cheers, Fabian > > > > Am Fr., 21. Feb. 2020 um 17:49 Uhr schrieb Rong Rong < > > walter...@gmail.com>: > > > > Congratulations Jingsong!! > > > > Cheers, > > Rong > > > > On Fri, Feb 21, 2020 at 8:45 AM Bowen Li wrote: > > > > Congrats, Jingsong! > > > > On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann > > > wrote: > > > > Congratulations Jingsong! > > > > Cheers, > > Till > > > > On Fri, Feb 21, 2020 at 4:03 PM Yun Gao > > wrote: > > > > Congratulations Jingsong! > > > > Best, > > Yun > > > > -- > > From:Jingsong Li > > Send Time:2020 Feb. 21 (Fri.) 21:42 > > To:Hequn Cheng > > Cc:Yang Wang ; Zhijiang < > > wangzhijiang...@aliyun.com>; Zhenghua Gao ; > > godfrey > > he ; dev ; user < > > u...@flink.apache.org> > > Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer > > > > Thanks everyone~ > > > > It's my pleasure to be part of the community. I hope I can make a > > better > > contribution in future. > > > > Best, > > Jingsong Lee > > > > On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng > > wrote: > > Congratulations Jingsong! Well deserved. > > > > Best, > > Hequn > > > > On Fri, Feb 21, 2020 at 2:42 PM Yang Wang > > wrote: > > Congratulations！Jingsong. Well deserved. > > > > > > Best, > > Yang > > > > Zhijiang 于2020年2月21日周五 下午1:18写道： > > Congrats Jingsong! Welcome on board! > > > > Best, > > Zhijiang > > > > -- > > From:Zhenghua Gao > > Send Time:2020 Feb. 21 (Fri.) 12:49 > > To:godfrey he > > Cc:dev ; user > > Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer > > > > Congrats Jingsong！ > > > > > > *Best Regards,* > > *Zhenghua Gao* > > > > > > On Fri, Feb 21, 2020 at 11:59 AM godfrey he > > wrote: > > Congrats Jingsong! Well deserved. > > > > Best, > > godfrey > > > > Jeff Zhang 于2020年2月21日周五 上午11:49写道： > > Congratulations！Jingsong. You deserve it > > > > wenlong.lwl 于2020年2月21日周五 上午11:43写道： > > Congrats Jingsong! > > > > On Fri, 21 Feb 2020 at 11:41, Dian Fu > > wrote: > > > > Congrats Jingsong! > > > > 在 2020年2月21日，上午11:39，Jark Wu 写道： > > > > Congratulations Jingsong! Well deserved. > > > > Best, > > Jark > > > > On Fri, 21 Feb 2020 at 11:32, zoudan wrote: > > > > Congratulations! Jingsong > > > > > > Best, > > Dan Zou > > > > > > > > > > > > -- > > Best Regards > > > > Jeff Zhang > > > > > > > > -- > > Best, Jingsong Lee > > > > > > > > > > > > > >
Re: Question about RocksDBStateBackend Compaction Filter state cleanup
Hi Lake Flink leverage RocksDB's background compaction mechanism to filter out-of-TTL entries (by comparing with current timestamp provided from RocksDB's time_provider) to not let them stay in newly compacted data. This would iterator over data entries with FlinkCompactionFilter::FilterV2 [1], and the parameter 'queryTimeAfterNumEntries' in Flink indicates the threshold 'query_time_after_num_entries_' in FrocksDB [2]. Once RocksDB iterator more than several entries .e.g 1000, it would call time_provider to update current timestamp to let the process of cleaning up more eagerly and accurately. [1] https://github.com/dataArtisans/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/utilities/flink/flink_compaction_filter.cc#L107 [2] https://github.com/dataArtisans/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/utilities/flink/flink_compaction_filter.h#L140 Best Yun Tang From: LakeShen Sent: Tuesday, March 17, 2020 15:30 To: dev ; user-zh ; user Subject: Question about RocksDBStateBackend Compaction Filter state cleanup Hi community , I see the flink RocksDBStateBackend state cleanup,now the code like this : StateTtlConfig ttlConfig = StateTtlConfig .newBuilder(Time.seconds(1)) .cleanupInRocksdbCompactFilter(1000) .build(); The default background cleanup for RocksDB backend queries the current timestamp each time 1000 entries have been processed. What's the meaning of 1000 entries? 1000 different key ? Thanks to your reply. Best regards, LakeShen
Re: [ANNOUNCE] New Committers and PMC member
Congratulations to all of you! Best Yun Tang From: Yang Wang Sent: Wednesday, April 1, 2020 22:28 To: dev Subject: Re: [ANNOUNCE] New Committers and PMC member Congratulations all. Best, Yang Leonard Xu 于2020年4月1日周三 下午10:15写道： > Congratulations Konstantin, Dawid and Zhijiang! Well deserved! > > Best, > Leonard Xu > > 在 2020年4月1日，21:22，Jark Wu 写道： > > > > Congratulations to you all! > > > > Best, > > Jark > > > > On Wed, 1 Apr 2020 at 20:33, Kurt Young wrote: > > > >> Congratulations to you all! > >> > >> Best, > >> Kurt > >> > >> > >> On Wed, Apr 1, 2020 at 7:41 PM Danny Chan wrote: > >> > >>> Congratulations! > >>> > >>> Best, > >>> Danny Chan > >>> 在 2020年4月1日 +0800 PM7:36，dev@flink.apache.org，写道： > >>>> > >>>> Congratulations! > >>> > >> > >
Configuring autolinks to Flink JIRA ticket in github repos
Hi community. I noticed that Github supports autolink reference recently [1]. This is helpful to allow developers could open Jira ticket link from pull requests title directly when accessing github repo. I have already created INFRA-20055 [2] to ask for configuration for seven Flink related github repos. Hope it could be resolved soon 🙂 [1] https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources [2] https://issues.apache.org/jira/browse/INFRA-20055 Best Yun Tang
Re: [ANNOUNCE] Apache Flink Stateful Functions 2.0.0 released
Excited to see the stateful functions release! Thanks for the great work of manager Gordon and everyone who ever contributed to this. Best Yun Tang From: Till Rohrmann Sent: Wednesday, April 8, 2020 14:30 To: dev Cc: Oytun Tez ; user Subject: Re: [ANNOUNCE] Apache Flink Stateful Functions 2.0.0 released Great news! Thanks a lot for being our release manager Gordon and to everyone who helped with the release. Cheers, Till On Wed, Apr 8, 2020 at 3:57 AM Congxian Qiu mailto:qcx978132...@gmail.com>> wrote: Thanks a lot for the release and your great job, Gordon! Also thanks to everyone who made this release possible! Best, Congxian Oytun Tez mailto:oy...@motaword.com>> 于2020年4月8日周三 上午2:55写道： > I should also add, I couldn't agree more with this sentence in the release > article: "state access/updates and messaging need to be integrated." > > This is something we strictly enforce in our Flink case, where we do not > refer to anything external for storage, use Flink as our DB. > > > > -- > > [image: MotaWord] > Oytun Tez > M O T A W O R D | CTO & Co-Founder > oy...@motaword.com<mailto:oy...@motaword.com> > > <https://www.motaword.com/blog> > > > On Tue, Apr 7, 2020 at 12:26 PM Oytun Tez > mailto:oy...@motaword.com>> wrote: > >> Great news! Thank you all. >> >> On Tue, Apr 7, 2020 at 12:23 PM Marta Paes Moreira >> mailto:ma...@ververica.com>> >> wrote: >> >>> Thank you for managing the release, Gordon ― you did a tremendous job! >>> And to everyone else who worked on pushing it through. >>> >>> Really excited about the new use cases that StateFun 2.0 unlocks for >>> Flink users and beyond! >>> >>> >>> Marta >>> >>> On Tue, Apr 7, 2020 at 4:47 PM Hequn Cheng >>> mailto:he...@apache.org>> wrote: >>> >>>> Thanks a lot for the release and your great job, Gordon! >>>> Also thanks to everyone who made this release possible! >>>> >>>> Best, >>>> Hequn >>>> >>>> On Tue, Apr 7, 2020 at 8:58 PM Tzu-Li (Gordon) Tai >>>> mailto:tzuli...@apache.org>> >>>> wrote: >>>> >>>>> The Apache Flink community is very happy to announce the release of >>>>> Apache Flink Stateful Functions 2.0.0. >>>>> >>>>> Stateful Functions is an API that simplifies building distributed >>>>> stateful applications. >>>>> It's based on functions with persistent state that can interact >>>>> dynamically with strong consistency guarantees. >>>>> >>>>> Please check out the release blog post for an overview of the release: >>>>> https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html >>>>> >>>>> The release is available for download at: >>>>> https://flink.apache.org/downloads.html >>>>> >>>>> Maven artifacts for Stateful Functions can be found at: >>>>> https://search.maven.org/search?q=g:org.apache.flink%20statefun >>>>> >>>>> Python SDK for Stateful Functions published to the PyPI index can be >>>>> found at: >>>>> https://pypi.org/project/apache-flink-statefun/ >>>>> >>>>> Official Docker image for building Stateful Functions applications is >>>>> currently being published to Docker Hub. >>>>> Dockerfiles for this release can be found at: >>>>> https://github.com/apache/flink-statefun-docker/tree/master/2.0.0 >>>>> Progress for creating the Docker Hub repository can be tracked at: >>>>> https://github.com/docker-library/official-images/pull/7749 >>>>> >>>>> The full release notes are available in Jira: >>>>> >>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346878 >>>>> >>>>> We would like to thank all contributors of the Apache Flink community >>>>> who made this release possible! >>>>> >>>>> Cheers, >>>>> Gordon >>>>> >>>> -- >> -- >> >> [image: MotaWord] >> Oytun Tez >> M O T A W O R D | CTO & Co-Founder >> oy...@motaword.com<mailto:oy...@motaword.com> >> >> <https://www.motaword.com/blog> >> >
[DISCUSS] Creating a new repo to host Flink benchmarks
Hi Flink devs, As Flink develops rapidly with more and more features added, how to ensure no performance regression existed has become more and more important. And we would like to create a new repo under apache project to host previous flink-benchmarks [1] repo, which is inspired when we discuss under FLINK-16850 [2] Some background context on flink-benchmarks, for those who are not familiar with the project yet: - Current flink-benchmarks does not align with the Flink release, which lead developers not easy to verify performance at specific Flink version because current flink-benchmarks always depends on the latest interfaces. - Above problem could be solved well if we could ensure flink-benchmarks also create release branch when we releasing Flink. However, current flink-benchmarks repo is hosted under dataArtisans (the former name of ververica) project, which is not involved in Flink release manual [3]. We propose to promote this repo under apache project so that release manager could have the right to release on flink-benchmarks. - The reason why we not involve flink-benchmarks into the apache/flink repo is because it heavily depends on JMH [4], which is under GPLv2 license. What do you think? Best, Yun Tang [1] https://github.com/dataArtisans/flink-benchmarks [2] https://issues.apache.org/jira/browse/FLINK-16850 [3] https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release [4] https://openjdk.java.net/projects/code-tools/jmh/
Re: Configuring autolinks to Flink JIRA ticket in github repos
Hi community The autolink to Flink JIRA ticket has taken effect. You could refer to the commit details page[1] to see all Flink JIRA titles within commits has the hyper link underline. Moreover, you don't need to use markdown language to create hyper link to Flink JIRA ticket when discussing in the pull requests. e.g FLINK-16850 could point to the link instead of [FLINK-16850](https://issues.apache.org/jira/browse/FLINK-16850) [1] https://github.com/apache/flink/commits/master Best Yun Tang From: Till Rohrmann Sent: Thursday, April 2, 2020 23:11 To: dev Subject: Re: Configuring autolinks to Flink JIRA ticket in github repos Nice, this is a cool feature. Thanks for asking INFRA for it. Cheers, Till On Wed, Apr 1, 2020 at 6:52 PM Yun Tang wrote: > Hi community. > > I noticed that Github supports autolink reference recently [1]. This is > helpful to allow developers could open Jira ticket link from pull requests > title directly when accessing github repo. > > I have already created INFRA-20055 [2] to ask for configuration for seven > Flink related github repos. Hope it could be resolved soon 🙂 > > > [1] > https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources > [2] https://issues.apache.org/jira/browse/INFRA-20055 > > Best > Yun Tang >
