RE: Flink Slow Execution

2024-01-18 Thread Qing Lim
multiple batch job throughout the day, I just initialize the cluster once at beginning of process and reuse it Best -Original Message- From: Jiabao Sun Sent: 18 January 2024 02:46 To: user@flink.apache.org Subject: RE: Flink Slow Execution Hi Dulce, MiniCluster is generally used for

RE: Flink Slow Execution

2024-01-17 Thread Jiabao Sun
Hi Dulce, MiniCluster is generally used for local testing and is limited by the resources of a single machine. When more tasks are executed, it may not be able to immediately acquire the resources needed to start the MiniCluster, resulting in slower startup times. If running Flink tasks in a

Flink Slow Execution

2024-01-17 Thread Dulce Morim
Hello, In a single JVM, I'm running multiple flink batch jobs locally using the MiniCluster (Java 17 and Flink 1.18). At the beginning of the process, the Mini Cluster starts pretty much instantly. However, I'm running into an issue where the more jobs I execute the longer the MiniCluster take