Re: [DISCUSS] Release new FRocksDB
Hi all, Thanks for everyone's advice, FRocksDB 6.20.3-ververica-2.0[1] had been released, compared with FRocksDB 6.20.3-ververica-1.0, there are the following 4 points of improvement: 1. [FLINK-30457] Add periodic_compaction_seconds option to RocksJava[2] 2. [FLINK-30321] Upgrade ZLIB of FRocksDB to 1.2.13[3] 3. PR 55: Avoid expensive ToString call when not in Debug[4] 4. [FLINK-24932] Support build FRocksDB Java on Apple silicon[5] Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86, MaxOS arm, and windows are included in FRocksDB 6.20.3-ververica-2.0 jar, many thanks to @Yun Tang for the work done to support the Linux ppc64le platform. And the FrocksDB in Flink 1.17 would be updated to 6.20.3-ververica-2.0, thanks again for everyone's suggestions and efforts. [1] https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-2.0 [2] https://issues.apache.org/jira/browse/FLINK-30457 [3] https://issues.apache.org/jira/browse/FLINK-30321 [4] https://github.com/ververica/frocksdb/pull/55 [5] https://issues.apache.org/jira/browse/FLINK-24932 Best, Yanfei ConradJam 于2022年12月20日周二 15:16写道: > > +1 , I agree to open a 7.x branch to study whether the new version of > rocksdb has brought something new changes > > Yun Tang 于2022年12月20日周二 12:47写道: > > > +1 to have a new FrocksDB release and thanks for the effors. > > > > For FLINK-24932, we planned to upgrade to RocksDB-6.29.x to support Apple > > ARM chips. However, we found another performance regression between 6.20.3 > > and 6.29.5 [1]. That’s why we still stay in the 6.20.3 branch. Since we > > already decided to create a new binary release, perhaps we could also try > > RocksDB-7.x to see the performance change. > > > > For the improvements to include in the new release, we can consider > > including [2] for better performance and FLINK-30457 [3] to support period > > compaction once we set TTL in Flink, which is asked by some users. > > > > > > [1] > > https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17557331=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17557331 > > [2] https://github.com/ververica/frocksdb/pull/55 > > [3] https://issues.apache.org/jira/browse/FLINK-30457 > > > > Best > > Yun Tang > > > > > > > > From: Yu Li > > Date: Monday, December 19, 2022 at 22:01 > > To: dev@flink.apache.org > > Subject: Re: [DISCUSS] Release new FRocksDB > > +1 to have a new FRocksDB release. Thanks Yanfei and all for the efforts! > > > > Best Regards, > > Yu > > > > > > On Fri, 16 Dec 2022 at 17:02, Martijn Visser > > wrote: > > > > > Hi Yanfei, > > > > > > Big +1 for adding ARM support. It would be great to be able to get tests > > to > > > pass on Mac M1s. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi > > > > > wrote: > > > > > > > Hi Yanfei, > > > > > > > > Makes sense, arm architecture support would be great. Thanks for > > looking > > > > into this. > > > > > > > > On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei > > wrote: > > > > > > > > > Hi devs, > > > > > > > > > > I'd like to bring up a discussion about releasing the new frocksdbjni > > > > > version, we are planning to adapt frocksdbjni to Apple M1 machines > > in > > > > > Flink 1.17[1], a new frocksdbjni jar needs to be released. > > > > > > > > > > Thanks to the efforts of the community, FLINK-24932[1] has been > > merged > > > > into > > > > > FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing > > a > > > > paid > > > > > CircleCI for testing, which ensures the correctness of the code. > > > > > > > > > > The new frocksdbjni version is still based on 6.20.3[2], the code > > > > > interfaces and performance[3] will not change. In addition to > > > > > FLINK-24932[1], some fixes of vulnerabilities[4] will also be > > included > > > > in > > > > > the new version, see FLINK-30321[5]. > > > > > > > > > > Are there any minor fixes you would like to include in this release? > > > > > Welcome to create PR for FrocksDB[6]. > > > > > Looking forward to your opinions. > > > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-24932 > > > > > [2] > > > > > > > > > > > > > > > > > > > https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0 > > > > > [3] > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889 > > > > > [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr > > > > > [5] https://issues.apache.org/jira/browse/FLINK-30321 > > > > > [6] https://github.com/ververica/frocksdb > > > > > > > > > > -- > > > > > Best, > > > > > Yanfei > > > > > > > > > > > > > >
Re: [DISCUSS] Release new FRocksDB
+1 , I agree to open a 7.x branch to study whether the new version of rocksdb has brought something new changes Yun Tang 于2022年12月20日周二 12:47写道: > +1 to have a new FrocksDB release and thanks for the effors. > > For FLINK-24932, we planned to upgrade to RocksDB-6.29.x to support Apple > ARM chips. However, we found another performance regression between 6.20.3 > and 6.29.5 [1]. That’s why we still stay in the 6.20.3 branch. Since we > already decided to create a new binary release, perhaps we could also try > RocksDB-7.x to see the performance change. > > For the improvements to include in the new release, we can consider > including [2] for better performance and FLINK-30457 [3] to support period > compaction once we set TTL in Flink, which is asked by some users. > > > [1] > https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17557331=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17557331 > [2] https://github.com/ververica/frocksdb/pull/55 > [3] https://issues.apache.org/jira/browse/FLINK-30457 > > Best > Yun Tang > > > > From: Yu Li > Date: Monday, December 19, 2022 at 22:01 > To: dev@flink.apache.org > Subject: Re: [DISCUSS] Release new FRocksDB > +1 to have a new FRocksDB release. Thanks Yanfei and all for the efforts! > > Best Regards, > Yu > > > On Fri, 16 Dec 2022 at 17:02, Martijn Visser > wrote: > > > Hi Yanfei, > > > > Big +1 for adding ARM support. It would be great to be able to get tests > to > > pass on Mac M1s. > > > > Best regards, > > > > Martijn > > > > On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi > > > wrote: > > > > > Hi Yanfei, > > > > > > Makes sense, arm architecture support would be great. Thanks for > looking > > > into this. > > > > > > On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei > wrote: > > > > > > > Hi devs, > > > > > > > > I'd like to bring up a discussion about releasing the new frocksdbjni > > > > version, we are planning to adapt frocksdbjni to Apple M1 machines > in > > > > Flink 1.17[1], a new frocksdbjni jar needs to be released. > > > > > > > > Thanks to the efforts of the community, FLINK-24932[1] has been > merged > > > into > > > > FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing > a > > > paid > > > > CircleCI for testing, which ensures the correctness of the code. > > > > > > > > The new frocksdbjni version is still based on 6.20.3[2], the code > > > > interfaces and performance[3] will not change. In addition to > > > > FLINK-24932[1], some fixes of vulnerabilities[4] will also be > included > > > in > > > > the new version, see FLINK-30321[5]. > > > > > > > > Are there any minor fixes you would like to include in this release? > > > > Welcome to create PR for FrocksDB[6]. > > > > Looking forward to your opinions. > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-24932 > > > > [2] > > > > > > > > > > > > > > https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0 > > > > [3] > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889 > > > > [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr > > > > [5] https://issues.apache.org/jira/browse/FLINK-30321 > > > > [6] https://github.com/ververica/frocksdb > > > > > > > > -- > > > > Best, > > > > Yanfei > > > > > > > > > >
Re: [DISCUSS] Release new FRocksDB
+1 to have a new FrocksDB release and thanks for the effors. For FLINK-24932, we planned to upgrade to RocksDB-6.29.x to support Apple ARM chips. However, we found another performance regression between 6.20.3 and 6.29.5 [1]. That’s why we still stay in the 6.20.3 branch. Since we already decided to create a new binary release, perhaps we could also try RocksDB-7.x to see the performance change. For the improvements to include in the new release, we can consider including [2] for better performance and FLINK-30457 [3] to support period compaction once we set TTL in Flink, which is asked by some users. [1] https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17557331=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17557331 [2] https://github.com/ververica/frocksdb/pull/55 [3] https://issues.apache.org/jira/browse/FLINK-30457 Best Yun Tang From: Yu Li Date: Monday, December 19, 2022 at 22:01 To: dev@flink.apache.org Subject: Re: [DISCUSS] Release new FRocksDB +1 to have a new FRocksDB release. Thanks Yanfei and all for the efforts! Best Regards, Yu On Fri, 16 Dec 2022 at 17:02, Martijn Visser wrote: > Hi Yanfei, > > Big +1 for adding ARM support. It would be great to be able to get tests to > pass on Mac M1s. > > Best regards, > > Martijn > > On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi > wrote: > > > Hi Yanfei, > > > > Makes sense, arm architecture support would be great. Thanks for looking > > into this. > > > > On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei wrote: > > > > > Hi devs, > > > > > > I'd like to bring up a discussion about releasing the new frocksdbjni > > > version, we are planning to adapt frocksdbjni to Apple M1 machines in > > > Flink 1.17[1], a new frocksdbjni jar needs to be released. > > > > > > Thanks to the efforts of the community, FLINK-24932[1] has been merged > > into > > > FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing a > > paid > > > CircleCI for testing, which ensures the correctness of the code. > > > > > > The new frocksdbjni version is still based on 6.20.3[2], the code > > > interfaces and performance[3] will not change. In addition to > > > FLINK-24932[1], some fixes of vulnerabilities[4] will also be included > > in > > > the new version, see FLINK-30321[5]. > > > > > > Are there any minor fixes you would like to include in this release? > > > Welcome to create PR for FrocksDB[6]. > > > Looking forward to your opinions. > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-24932 > > > [2] > > > > > > > > > https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0 > > > [3] > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889 > > > [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr > > > [5] https://issues.apache.org/jira/browse/FLINK-30321 > > > [6] https://github.com/ververica/frocksdb > > > > > > -- > > > Best, > > > Yanfei > > > > > >
Re: [DISCUSS] Release new FRocksDB
+1 to have a new FRocksDB release. Thanks Yanfei and all for the efforts! Best Regards, Yu On Fri, 16 Dec 2022 at 17:02, Martijn Visser wrote: > Hi Yanfei, > > Big +1 for adding ARM support. It would be great to be able to get tests to > pass on Mac M1s. > > Best regards, > > Martijn > > On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi > wrote: > > > Hi Yanfei, > > > > Makes sense, arm architecture support would be great. Thanks for looking > > into this. > > > > On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei wrote: > > > > > Hi devs, > > > > > > I'd like to bring up a discussion about releasing the new frocksdbjni > > > version, we are planning to adapt frocksdbjni to Apple M1 machines in > > > Flink 1.17[1], a new frocksdbjni jar needs to be released. > > > > > > Thanks to the efforts of the community, FLINK-24932[1] has been merged > > into > > > FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing a > > paid > > > CircleCI for testing, which ensures the correctness of the code. > > > > > > The new frocksdbjni version is still based on 6.20.3[2], the code > > > interfaces and performance[3] will not change. In addition to > > > FLINK-24932[1], some fixes of vulnerabilities[4] will also be included > > in > > > the new version, see FLINK-30321[5]. > > > > > > Are there any minor fixes you would like to include in this release? > > > Welcome to create PR for FrocksDB[6]. > > > Looking forward to your opinions. > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-24932 > > > [2] > > > > > > > > > https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0 > > > [3] > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889 > > > [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr > > > [5] https://issues.apache.org/jira/browse/FLINK-30321 > > > [6] https://github.com/ververica/frocksdb > > > > > > -- > > > Best, > > > Yanfei > > > > > >
Re: [DISCUSS] Release new FRocksDB
Hi Yanfei, Big +1 for adding ARM support. It would be great to be able to get tests to pass on Mac M1s. Best regards, Martijn On Mon, Dec 12, 2022 at 3:37 PM Márton Balassi wrote: > Hi Yanfei, > > Makes sense, arm architecture support would be great. Thanks for looking > into this. > > On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei wrote: > > > Hi devs, > > > > I'd like to bring up a discussion about releasing the new frocksdbjni > > version, we are planning to adapt frocksdbjni to Apple M1 machines in > > Flink 1.17[1], a new frocksdbjni jar needs to be released. > > > > Thanks to the efforts of the community, FLINK-24932[1] has been merged > into > > FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing a > paid > > CircleCI for testing, which ensures the correctness of the code. > > > > The new frocksdbjni version is still based on 6.20.3[2], the code > > interfaces and performance[3] will not change. In addition to > > FLINK-24932[1], some fixes of vulnerabilities[4] will also be included > in > > the new version, see FLINK-30321[5]. > > > > Are there any minor fixes you would like to include in this release? > > Welcome to create PR for FrocksDB[6]. > > Looking forward to your opinions. > > > > [1] https://issues.apache.org/jira/browse/FLINK-24932 > > [2] > > > > > https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0 > > [3] > > > > > https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889 > > [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr > > [5] https://issues.apache.org/jira/browse/FLINK-30321 > > [6] https://github.com/ververica/frocksdb > > > > -- > > Best, > > Yanfei > > >
Re: [DISCUSS] Release new FRocksDB
Hi Yanfei, Makes sense, arm architecture support would be great. Thanks for looking into this. On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei wrote: > Hi devs, > > I'd like to bring up a discussion about releasing the new frocksdbjni > version, we are planning to adapt frocksdbjni to Apple M1 machines in > Flink 1.17[1], a new frocksdbjni jar needs to be released. > > Thanks to the efforts of the community, FLINK-24932[1] has been merged into > FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing a paid > CircleCI for testing, which ensures the correctness of the code. > > The new frocksdbjni version is still based on 6.20.3[2], the code > interfaces and performance[3] will not change. In addition to > FLINK-24932[1], some fixes of vulnerabilities[4] will also be included in > the new version, see FLINK-30321[5]. > > Are there any minor fixes you would like to include in this release? > Welcome to create PR for FrocksDB[6]. > Looking forward to your opinions. > > [1] https://issues.apache.org/jira/browse/FLINK-24932 > [2] > > https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0 > [3] > > https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889 > [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr > [5] https://issues.apache.org/jira/browse/FLINK-30321 > [6] https://github.com/ververica/frocksdb > > -- > Best, > Yanfei >
[DISCUSS] Release new FRocksDB
Hi devs, I'd like to bring up a discussion about releasing the new frocksdbjni version, we are planning to adapt frocksdbjni to Apple M1 machines in Flink 1.17[1], a new frocksdbjni jar needs to be released. Thanks to the efforts of the community, FLINK-24932[1] has been merged into FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing a paid CircleCI for testing, which ensures the correctness of the code. The new frocksdbjni version is still based on 6.20.3[2], the code interfaces and performance[3] will not change. In addition to FLINK-24932[1], some fixes of vulnerabilities[4] will also be included in the new version, see FLINK-30321[5]. Are there any minor fixes you would like to include in this release? Welcome to create PR for FrocksDB[6]. Looking forward to your opinions. [1] https://issues.apache.org/jira/browse/FLINK-24932 [2] https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0 [3] https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889 [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr [5] https://issues.apache.org/jira/browse/FLINK-30321 [6] https://github.com/ververica/frocksdb -- Best, Yanfei