This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13015-379292194cc9ee9ecba2f6adf07cefcc94a7db21 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit e800766f2fdc394824c055e962d1ee362d881531 Author: Doyeon Kim <[email protected]> AuthorDate: Tue Nov 10 15:09:22 2026 +0000 [Improve][Zeta] Exclude engine_checkpoint_monitor from IMap persistence (#13015) --- docs/en/engines/zeta/hybrid-cluster-deployment.md | 10 ++-- .../engines/zeta/separated-cluster-deployment.md | 10 ++-- .../en/getting-started/kubernetes/configuration.md | 12 +---- docs/zh/engines/zeta/hybrid-cluster-deployment.md | 9 ++-- .../engines/zeta/separated-cluster-deployment.md | 9 ++-- .../zh/getting-started/kubernetes/configuration.md | 12 +---- .../engine/e2e/ClusterFaultToleranceIT.java | 18 +++++++ .../engine/server/persistence/FileMapStore.java | 21 ++++++-- .../server/persistence/FileMapStoreTest.java | 63 +++++++++++++++++++--- 9 files changed, 118 insertions(+), 46 deletions(-) diff --git a/docs/en/engines/zeta/hybrid-cluster-deployment.md b/docs/en/engines/zeta/hybrid-cluster-deployment.md index 87a3deb7c3..959087969f 100644 --- a/docs/en/engines/zeta/hybrid-cluster-deployment.md +++ b/docs/en/engines/zeta/hybrid-cluster-deployment.md @@ -338,10 +338,12 @@ map: fs.defaultFS: file:/// ``` -Note: `engine_runningJobMetrics` stores high-frequency runtime metrics snapshots and is -intentionally excluded from persistent IMAP storage even when `map.engine*` uses `map-store`. This -avoids excessive WAL growth for observability-only state. After an engine restart, running-job -metrics are rebuilt from subsequent reports instead of continuing from the pre-restart snapshot. +Note: `engine_runningJobMetrics` stores high-frequency runtime metrics snapshots and +`engine_checkpoint_monitor` stores the checkpoint overview and history served by the REST API. +Both are intentionally excluded from persistent IMAP storage even when `map.engine*` uses +`map-store`. This avoids excessive WAL growth for observability-only state. After an engine +restart, running-job metrics are rebuilt from subsequent reports and the checkpoint overview and +history start again from the next checkpoint, instead of continuing from the pre-restart state. If using OSS, you can configure it as follows: diff --git a/docs/en/engines/zeta/separated-cluster-deployment.md b/docs/en/engines/zeta/separated-cluster-deployment.md index bcc20d1393..f15dc52635 100644 --- a/docs/en/engines/zeta/separated-cluster-deployment.md +++ b/docs/en/engines/zeta/separated-cluster-deployment.md @@ -290,10 +290,12 @@ map: fs.defaultFS: file:/// ``` -Note: `engine_runningJobMetrics` stores high-frequency runtime metrics snapshots and is -intentionally excluded from persistent IMAP storage even when `map.engine*` uses `map-store`. This -avoids excessive WAL growth for observability-only state. After an engine restart, running-job -metrics are rebuilt from subsequent reports instead of continuing from the pre-restart snapshot. +Note: `engine_runningJobMetrics` stores high-frequency runtime metrics snapshots and +`engine_checkpoint_monitor` stores the checkpoint overview and history served by the REST API. +Both are intentionally excluded from persistent IMAP storage even when `map.engine*` uses +`map-store`. This avoids excessive WAL growth for observability-only state. After an engine +restart, running-job metrics are rebuilt from subsequent reports and the checkpoint overview and +history start again from the next checkpoint, instead of continuing from the pre-restart state. If you use OSS, you can configure it like this: diff --git a/docs/en/getting-started/kubernetes/configuration.md b/docs/en/getting-started/kubernetes/configuration.md index 94a9888866..b94aeefaf4 100644 --- a/docs/en/getting-started/kubernetes/configuration.md +++ b/docs/en/getting-started/kubernetes/configuration.md @@ -180,20 +180,12 @@ MapStore uses `FileMapStoreFactory`. The `type: hdfs` entry is the identifier of ### Checkpoint Monitor MapStore -If you do not want to persist checkpoint monitor data, add an explicit override for `engine_checkpoint_monitor`: - -```yaml -hazelcast: - map: - engine_checkpoint_monitor: - map-store: - enabled: false -``` +`engine_checkpoint_monitor` is never written to the MapStore, even when `engine*` enables it, so it needs no override. :::info Note This IMap stores checkpoint overview and history data used by REST APIs and UI observability. It is not the authoritative checkpoint recovery state. Checkpoint recovery state is written by `seatunnel.engine.checkpoint.storage` to HDFS, S3, OSS, or another configured checkpoint storage backend. -This override is optional. Use it when you want to avoid persisting observability-only data and reduce MapStore/WAL write amplification. Omit it if you want checkpoint monitor overview/history to use the same MapStore settings as other `engine*` IMaps. +Like `engine_runningJobMetrics`, it is excluded from MapStore persistence to avoid MapStore/WAL write amplification. After every member restarts, the checkpoint overview and history start again from the next checkpoint. An existing `engine_checkpoint_monitor` override with `enabled: false` is no longer needed and can be removed. ::: ### HDFS diff --git a/docs/zh/engines/zeta/hybrid-cluster-deployment.md b/docs/zh/engines/zeta/hybrid-cluster-deployment.md index b59101405f..58de25327d 100644 --- a/docs/zh/engines/zeta/hybrid-cluster-deployment.md +++ b/docs/zh/engines/zeta/hybrid-cluster-deployment.md @@ -334,10 +334,11 @@ map: fs.defaultFS: file:/// ``` -说明:`engine_runningJobMetrics` 保存的是高频运行时指标快照。即使通过 `map.engine*` -配置了 `map-store`,它也会被有意排除在持久化 IMAP 存储之外,以避免仅用于可观测性的状态导致 -WAL 持续膨胀。Engine 重启后,running-job metrics 不会延续重启前的 snapshot,而是由后续 -report 重新构建。 +说明:`engine_runningJobMetrics` 保存的是高频运行时指标快照,`engine_checkpoint_monitor` +保存的是 REST API 使用的 checkpoint overview/history。即使通过 `map.engine*` 配置了 +`map-store`,它们也会被有意排除在持久化 IMAP 存储之外,以避免仅用于可观测性的状态导致 WAL +持续膨胀。Engine 重启后,running-job metrics 由后续 report 重新构建,checkpoint +overview/history 从下一次 checkpoint 开始重新记录,而不会延续重启前的状态。 如果您使用 OSS,可以像这样配置: diff --git a/docs/zh/engines/zeta/separated-cluster-deployment.md b/docs/zh/engines/zeta/separated-cluster-deployment.md index 311b0d4f01..e78b5f9645 100644 --- a/docs/zh/engines/zeta/separated-cluster-deployment.md +++ b/docs/zh/engines/zeta/separated-cluster-deployment.md @@ -295,10 +295,11 @@ map: fs.defaultFS: file:/// ``` -说明:`engine_runningJobMetrics` 保存的是高频运行时指标快照。即使通过 `map.engine*` -配置了 `map-store`,它也会被有意排除在持久化 IMAP 存储之外,以避免仅用于可观测性的状态导致 -WAL 持续膨胀。Engine 重启后,running-job metrics 不会延续重启前的 snapshot,而是由后续 -report 重新构建。 +说明:`engine_runningJobMetrics` 保存的是高频运行时指标快照,`engine_checkpoint_monitor` +保存的是 REST API 使用的 checkpoint overview/history。即使通过 `map.engine*` 配置了 +`map-store`,它们也会被有意排除在持久化 IMAP 存储之外,以避免仅用于可观测性的状态导致 WAL +持续膨胀。Engine 重启后,running-job metrics 由后续 report 重新构建,checkpoint +overview/history 从下一次 checkpoint 开始重新记录,而不会延续重启前的状态。 如果您使用 OSS,可以像这样配置: diff --git a/docs/zh/getting-started/kubernetes/configuration.md b/docs/zh/getting-started/kubernetes/configuration.md index 35a1e210df..2f961a22c9 100644 --- a/docs/zh/getting-started/kubernetes/configuration.md +++ b/docs/zh/getting-started/kubernetes/configuration.md @@ -180,20 +180,12 @@ MapStore 使用 `FileMapStoreFactory`。其中 `type: hdfs` 是 IMap 文件存 ### Checkpoint Monitor MapStore -如果不希望持久化 checkpoint monitor 数据,可以为 `engine_checkpoint_monitor` 添加显式覆盖: - -```yaml -hazelcast: - map: - engine_checkpoint_monitor: - map-store: - enabled: false -``` +即使 `engine*` 启用了 MapStore,`engine_checkpoint_monitor` 也不会写入 MapStore,因此无需为它添加覆盖。 :::info 说明 这个 IMap 存储的是 REST API 和 UI 观测使用的 checkpoint overview/history 数据,并不是用于恢复的权威 checkpoint 状态。真正的 checkpoint 恢复状态由 `seatunnel.engine.checkpoint.storage` 写入 HDFS、S3、OSS 或其他已配置的 checkpoint 存储后端。 -这个覆盖是可选的。需要避免持久化仅用于观测的数据、减少 MapStore/WAL 写放大时可以使用;如果希望 checkpoint monitor 的 overview/history 与其他 `engine*` IMap 使用相同的 MapStore 配置,可以不添加该覆盖。 +与 `engine_runningJobMetrics` 一样,它被排除在 MapStore 持久化之外,以避免 MapStore/WAL 写放大。所有节点重启后,checkpoint overview/history 会从下一次 checkpoint 开始重新记录。已有的 `enabled: false` 的 `engine_checkpoint_monitor` 覆盖不再需要,可以删除。 ::: ### HDFS diff --git a/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/ClusterFaultToleranceIT.java b/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/ClusterFaultToleranceIT.java index f99cca0950..7f352753ce 100644 --- a/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/ClusterFaultToleranceIT.java +++ b/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/ClusterFaultToleranceIT.java @@ -31,6 +31,7 @@ import org.apache.seatunnel.engine.common.config.JobConfig; import org.apache.seatunnel.engine.common.config.SeaTunnelConfig; import org.apache.seatunnel.engine.common.job.JobStatus; import org.apache.seatunnel.engine.server.SeaTunnelServerStarter; +import org.apache.seatunnel.engine.server.common.statestore.EngineStateStoreNames; import org.awaitility.Awaitility; import org.junit.jupiter.api.Assertions; @@ -42,6 +43,7 @@ import com.hazelcast.client.config.ClientConfig; import com.hazelcast.config.Config; import com.hazelcast.core.EntryView; import com.hazelcast.instance.impl.HazelcastInstanceImpl; +import com.hazelcast.map.IMap; import lombok.NonNull; import lombok.extern.slf4j.Slf4j; @@ -849,6 +851,22 @@ public class ClusterFaultToleranceIT { Assertions.assertTrue(lineNumberFromDir > 1); }); + // engine_checkpoint_monitor matches the engine* map-store config, but it only holds + // observability state: it keeps the job's checkpoint overview in memory and writes + // nothing to the IMap storage, next to the engine maps that are persisted. + IMap<Long, Object> checkpointMonitor = + finalNode.getMap(EngineStateStoreNames.CHECKPOINT_MONITOR); + Awaitility.await() + .atMost(60, TimeUnit.SECONDS) + .untilAsserted( + () -> Assertions.assertTrue(checkpointMonitor.containsKey(jobId))); + File imapStorageRoot = + new File("/tmp/seatunnel/imap", TestUtils.getClusterName(testClusterName)); + Assertions.assertTrue( + new File(imapStorageRoot, Constant.IMAP_RUNNING_JOB_STATE).isDirectory()); + Assertions.assertFalse( + new File(imapStorageRoot, EngineStateStoreNames.CHECKPOINT_MONITOR).exists()); + Thread.sleep(5000); // shutdown all node node1.shutdown(); diff --git a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/persistence/FileMapStore.java b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/persistence/FileMapStore.java index 16c20e9330..8592f4131d 100644 --- a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/persistence/FileMapStore.java +++ b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/persistence/FileMapStore.java @@ -45,6 +45,9 @@ import java.util.Properties; * entry. Hazelcast does not delete an entry from the map store when it expires, so the expiration * time is stored with the entry and handed back on load; otherwise an entry put with a TTL would be * loaded again after a restart and never expire. + * + * <p>The auxiliary observability maps, {@link EngineStateStoreNames#RUNNING_JOB_METRICS} and {@link + * EngineStateStoreNames#CHECKPOINT_MONITOR}, are never persisted, whatever the map-store config. */ @Slf4j public class FileMapStore implements EntryStore<Object, Object>, MapLoaderLifecycleSupport { @@ -53,11 +56,11 @@ public class FileMapStore implements EntryStore<Object, Object>, MapLoaderLifecy @Override public void init(HazelcastInstance hazelcastInstance, Properties properties, String mapName) { - if (EngineStateStoreNames.RUNNING_JOB_METRICS.equals(mapName)) { + if (isAuxiliaryObservabilityMap(mapName)) { this.mapStorage = NoOpMapStorage.INSTANCE; log.info( - "Skip persistence for map '{}' because runtime metrics snapshots are auxiliary " - + "observability state and should not write to persistent IMAP storage.", + "Skip persistence for map '{}' because it holds auxiliary observability state " + + "and should not write to persistent IMAP storage.", mapName); return; } @@ -148,6 +151,18 @@ public class FileMapStore implements EntryStore<Object, Object>, MapLoaderLifecy return mapStorage.loadAllKeys(); } + /** + * Returns whether the map only holds observability state that nothing on the recovery path + * reads. Running jobs refill these maps after a restart: workers report runtime metrics again, + * and the checkpoint monitor records the next checkpoint. Persisting them only grows the + * append-only storage; the checkpoint monitor alone rewrites a job's whole overview on every + * checkpoint trigger, subtask acknowledgement and completion. + */ + private static boolean isAuxiliaryObservabilityMap(String mapName) { + return EngineStateStoreNames.RUNNING_JOB_METRICS.equals(mapName) + || EngineStateStoreNames.CHECKPOINT_MONITOR.equals(mapName); + } + private static final class NoOpMapStorage implements IMapStorage { private static final NoOpMapStorage INSTANCE = new NoOpMapStorage(); diff --git a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/persistence/FileMapStoreTest.java b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/persistence/FileMapStoreTest.java index d6d2491fdb..53911eced3 100644 --- a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/persistence/FileMapStoreTest.java +++ b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/persistence/FileMapStoreTest.java @@ -26,6 +26,8 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.api.condition.EnabledOnOs; import org.junit.jupiter.api.condition.OS; import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; import com.hazelcast.config.Config; import com.hazelcast.config.JoinConfig; @@ -37,8 +39,11 @@ import com.hazelcast.core.HazelcastInstance; import com.hazelcast.map.EntryLoader.MetadataAwareValue; import com.hazelcast.map.IMap; +import java.nio.file.Files; import java.nio.file.Path; +import java.util.Arrays; import java.util.Collections; +import java.util.List; import java.util.Properties; import java.util.concurrent.TimeUnit; @@ -114,18 +119,19 @@ public class FileMapStoreTest { "Exception message should identify the unknown type. Got: " + msg); } - @Test - public void testMetricsSnapshotMapSkipsPersistentStorageInitialization() { + @ParameterizedTest + @ValueSource( + strings = { + EngineStateStoreNames.RUNNING_JOB_METRICS, + EngineStateStoreNames.CHECKPOINT_MONITOR + }) + public void testAuxiliaryObservabilityMapSkipsPersistentStorageInitialization(String mapName) { FileMapStore store = new FileMapStore(); Properties props = new Properties(); props.setProperty("type", "non-existent-storage-type"); Assertions.assertDoesNotThrow( - () -> - store.init( - mock(HazelcastInstance.class), - props, - EngineStateStoreNames.RUNNING_JOB_METRICS)); + () -> store.init(mock(HazelcastInstance.class), props, mapName)); Assertions.assertEquals(Collections.emptySet(), store.loadAllKeys()); Assertions.assertDoesNotThrow(() -> store.store("k", new MetadataAwareValue<>("v"))); Assertions.assertDoesNotThrow(() -> store.delete("k")); @@ -133,6 +139,49 @@ public class FileMapStoreTest { store.destroy(); } + /** + * The auxiliary observability maps match the {@code engine*} map-store config like every other + * engine map. Verifies that they are not written to the IMap storage and so start empty after a + * restart, while an engine state map next to them is still persisted and loaded back. + */ + @Test + public void testAuxiliaryObservabilityMapsAreNotPersistedAcrossRestart() { + String clusterName = "file-map-store-auxiliary-" + System.nanoTime(); + String persistedMapName = Constant.IMAP_RUNNING_JOB_STATE; + List<String> auxiliaryMapNames = + Arrays.asList( + EngineStateStoreNames.RUNNING_JOB_METRICS, + EngineStateStoreNames.CHECKPOINT_MONITOR); + + HazelcastInstance writer = Hazelcast.newHazelcastInstance(hazelcastConfig(clusterName)); + try { + writer.getMap(persistedMapName).put(1L, "persisted"); + auxiliaryMapNames.forEach(mapName -> writer.getMap(mapName).put(1L, "auxiliary")); + } finally { + writer.shutdown(); + } + + Path storageRoot = tempDir.resolve("test-cluster"); + Assertions.assertTrue(Files.isDirectory(storageRoot.resolve(persistedMapName))); + auxiliaryMapNames.forEach( + mapName -> + Assertions.assertFalse( + Files.exists(storageRoot.resolve(mapName)), + mapName + " must not be written to the IMap storage")); + + HazelcastInstance restarted = Hazelcast.newHazelcastInstance(hazelcastConfig(clusterName)); + try { + Assertions.assertEquals("persisted", restarted.getMap(persistedMapName).get(1L)); + auxiliaryMapNames.forEach( + mapName -> + Assertions.assertTrue( + restarted.getMap(mapName).isEmpty(), + mapName + " must not be loaded again after a restart")); + } finally { + restarted.shutdown(); + } + } + /** * Hazelcast does not delete an entry from the map store when its TTL expires. Verifies that * after a restart an expired entry is not loaded again, and an entry put with a TTL keeps its
