Anyone encounters the non-heap memory increasing issue when using RocksDB as state backend and has better solutions?

2023-12-28 Thread Sue Alen
Does anyone encounter the non-heap memory increasing issue when using RocksDB as state backend with memory controlling enabled? It has plagued us for long time. Our flink jobs are deployed under Kubernetes, and they would be killed because of OOM by the increasing non-heap memory by Kubernetes.

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2023-12-28 Thread Zakelly Lan
Hi everyone, Thanks all for your comments! As many of you have questions about the names for boolean options, I suggest we make a naming rule for them. For now I could think of three options: Option 1: Use enumeration options if possible. But this may cause some name collisions or confusion as

Re: [DISCUSS] FLIP-398: Improve Serialization Configuration And Usage In Flink

2023-12-28 Thread Ken Krugler
Hi Xintong, I agree that decoupling from Kryo is a bigger topic, well beyond the scope of this FLIP. The reason I’d brought up Fury is that this increases my confidence that Flink will want to decouple from Kryo sooner rather than later. So I feel it would be worth investing in a (minor) name

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2023-12-28 Thread Xuannan Su
Hi Zakelly, Thanks for driving this! The organization of the configuration option in the FLIP looks much cleaner and easier to understand. +1 to the FLIP. Just some questions from me. 1. I think the change to the ConfigOptions should be put in the `Public Interface` section, instead of

[jira] [Created] (FLINK-33958) Implement restore tests for IntervalJoin node

2023-12-28 Thread Bonnie Varghese (Jira)
Bonnie Varghese created FLINK-33958: --- Summary: Implement restore tests for IntervalJoin node Key: FLINK-33958 URL: https://issues.apache.org/jira/browse/FLINK-33958 Project: Flink Issue

[jira] [Created] (FLINK-33957) Generic Log-Based Incremental Checkpoint fails when Task Local Recovery is enabled

2023-12-28 Thread Prabhu Joseph (Jira)
Prabhu Joseph created FLINK-33957: - Summary: Generic Log-Based Incremental Checkpoint fails when Task Local Recovery is enabled Key: FLINK-33957 URL: https://issues.apache.org/jira/browse/FLINK-33957

Re: [DISCUSS] FLIP-403: High Availability Services for OLAP Scenarios

2023-12-28 Thread Yangze Guo
Thanks for the response, Zhanghao. PersistenceServices sounds good to me. Best, Yangze Guo On Wed, Dec 27, 2023 at 11:30 AM Zhanghao Chen wrote: > > Thanks for driving this effort, Yangze! The proposal overall LGTM. Other from > the throughput enhancement in the OLAP scenario, the separation

Re: Building Flink JDBC connector locally on M1 Mac

2023-12-28 Thread Martijn Visser
Hi David, I believe this is because the testcontainers rely on the log4j configuration settings. Github CI runs something like mvn clean install -Dlog4j.configurationFile=tools/ci/log4j.properties Best regards, Martijn On Fri, Dec 8, 2023 at 5:30 PM David Radley wrote: > > Hi, > When I build

[jira] [Created] (FLINK-33956) Bump org.apache.zookeeper:zookeeper from 3.7.1 to 3.7.2

2023-12-28 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-33956: -- Summary: Bump org.apache.zookeeper:zookeeper from 3.7.1 to 3.7.2 Key: FLINK-33956 URL: https://issues.apache.org/jira/browse/FLINK-33956 Project: Flink

Re: [VOTE] Release 1.18.1, release candidate #2

2023-12-28 Thread Yun Tang
Thanks Jing for driving this release. +1 (non-binding) * Download artifacts and verify the signatures. * Verified the web PR * Verified the number of Python packages is 11 * Started a local cluster and verified FLIP-291 to see the rescale results. * Verified the jar packages were

Re: [DISCUSS] FLIP-399: Flink Connector Doris

2023-12-28 Thread Martijn Visser
+1 for this :) On Thu, Dec 7, 2023 at 6:05 AM wudi <676366...@qq.com.invalid> wrote: > > > Hi all, > > As discussed in the previous email [1], about contributing the Flink Doris > Connector to the Flink community. > > > Apache Doris[2] is a high-performance, real-time analytical database based

Re: [VOTE] Release flink-shaded 18.0, release candidate #1

2023-12-28 Thread Martijn Visser
Hi, +1 (binding) - Validated hashes - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven - Verified licenses - Verified web PRs Best regards, Martijn On Mon, Dec 11, 2023 at 12:11 AM Sergey Nuyanzin wrote: > > Hey everyone, > > The vote

Re: [NOTICE] Hive connector externalization

2023-12-28 Thread Martijn Visser
Hi Sergey, Is the next step that we need to generate a release of the externalized code? Did someone already volunteer for that? Best regards, Martijn On Mon, Dec 11, 2023 at 3:00 AM yuxia wrote: > > Thanks Sergey for the work. Happy to see we can externalize Hive connector > finally. > >

[DISCUSS] Hadoop 2 vs Hadoop 3 usage

2023-12-28 Thread Martijn Visser
Hi all, I want to get some insights on how many users are still using Hadoop 2 vs how many users are using Hadoop 3. Flink currently requires a minimum version of Hadoop 2.10.2 for certain features, but also extensively uses Hadoop 3 (like for the file system implementations) Hadoop 2 has a

Re: [DISCUSS] FLIP-402: Extend ZooKeeper Curator configurations

2023-12-28 Thread Martijn Visser
Hi Oleksandr, The FLIP talks about Curator, but outside of flink test utils, the usage of Curator is only for test purposes. I don't think there's anything preventing you right now from providing these additional parameters as values in the flink-conf.yaml ? Best regards, Martijn On Thu, Dec

[jira] [Created] (FLINK-33955) UnsupportedFileSystemException when trying to save data to Azure's abfss File System

2023-12-28 Thread Jira
Alek Łańduch created FLINK-33955: Summary: UnsupportedFileSystemException when trying to save data to Azure's abfss File System Key: FLINK-33955 URL: https://issues.apache.org/jira/browse/FLINK-33955

[ANNOUNCE] Apache Flink Pulsar Connector 4.1.0 released

2023-12-28 Thread Leonard Xu
The Apache Flink community is very happy to announce the release of Apache Flink Pulsar Connector 4.1.0 which is compatible with the Apache Flink 1.17.x and 1.18.x release series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available,

Re: [VOTE] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-28 Thread Timo Walther
+1 (binding) Cheers, Timo > Am 28.12.2023 um 03:13 schrieb Yuepeng Pan : > > +1 (non-binding). > > Best, > Yuepeng Pan. > > > > > At 2023-12-28 09:19:37, "Lincoln Lee" wrote: >> +1 (binding) >> >> Best, >> Lincoln Lee >> >> >> Martijn Visser 于2023年12月27日周三 23:16写道: >> >>> +1