w3ll1ngt commented on code in PR #13741: URL: https://github.com/apache/ignite/pull/13741#discussion_r4454667303
########## docs/_docs/server-node-file-system-layout.adoc: ########## @@ -0,0 +1,251 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + += Server Node File System Layout +:javaFile: {javaCodeDir}/PersistenceTuning.java + +This page brings together recommendations for placing server node files. +Use it to plan disk space, manage retained files, and protect node storage from log and diagnostic growth. + +These recommendations are intended for production server nodes. +On nodes without native persistence, the guidance for the work directory, logs, and JVM diagnostics still applies. + +== Separate File Systems + +Use separate file systems to keep log and diagnostic growth from taking space needed by the node. +For workloads with heavy writes, placing data files and the active WAL on separate devices can also improve write performance. +Choose a layout that fits your workload and available storage. +If several file types share a file system, plan for their combined use, limit retained logs and diagnostic files, and leave free space for checkpoints and WAL rotation. + +Running out of disk space can stop a node and prevent it from restarting. + +A separate file system bounds capacity: directories placed on the same file system share its free space. +It does not by itself reduce I/O contention, which requires a separate device or independently provisioned throughput. +Partitions of one physical disk isolate capacity but still share the failure risk of that disk. + +== Recommended Placement + +[cols="1,1,3", options="header"] +|=== +|Files |Configuration |Placement + +|Data and indexes +|`DataStorageConfiguration.storagePath` and `DataStorageConfiguration.extraStoragePaths` +|Dedicated, durable data file systems. Keep logs, diagnostic files, and retained backups off them, so that unrelated growth does not take the space needed by checkpoint writes and recovery. + +|Active WAL +|`DataStorageConfiguration.walPath` +|A separate file system with room for the active segments and a rotation reserve, see <<Sizing the WAL Volume>>. See <<Configuration Example>> for the path setting. On write-heavy workloads, a separate device for the active WAL and the data files reduces the competition between checkpoint page writes and WAL writes. + +|WAL archive +|`DataStorageConfiguration.walArchivePath` +|Preferably a separate file system. Plan its capacity for the retained segments, see <<Sizing the WAL Archive>>, and for the features that share it: CDC and incremental snapshots, see <<Account for Hard Links>>. + +|CDC +|`DataStorageConfiguration.cdcWalPath` +|The *same file system as the WAL archive*, because CDC uses hard links to archived WAL segments. A separate CDC directory does not isolate its disk-space usage. Bound its retention with `cdcWalDirectoryMaxSize`. + +|Work directory +|`IgniteConfiguration.workDirectory` +|Durable storage with free space to spare. The directory keeps binary metadata, marshaller mappings, and other node files even when data and WAL are configured elsewhere. Keep it apart from logs and diagnostic output, and do not treat it as disposable temporary storage. + +|Snapshots +|`IgniteConfiguration.snapshotPath` and `DataStorageConfiguration.extraSnapshotPaths` +|For full snapshots without increments, preferably a separate file system and, where available, separate storage: the snapshot adds a second read and write pass over the data. For incremental snapshots, the snapshot root must be on the same file system as the WAL archive, see <<Account for Hard Links>>. Keep independent backup copies outside the node storage. + +|Ignite logs +|Log4j2 appenders in the logging configuration; `java.util.logging.properties` and the `IGNITE_LOG_DIR` environment variable for the built-in JUL logger +|A separate file system, with a bound on the total retained volume rather than on individual files. When logs share a file system with data, their growth can take the space that checkpoints and WAL rotation need. `IGNITE_LOG_DIR` affects only the built-in JUL logger, not Log4j2 appenders. + +|JVM and OS diagnostics +|GC logging options; JFR repository and output settings; heap and core dump paths; monitoring-tool output +|Storage separate from data, WAL, and the work directory, with retained output bounded and space reserved for large dumps, see <<Logs and Diagnostic Output>>. +|=== + +If some roles have to share a file system, size it for their combined usage and limit what each role retains. +Keep Ignite files and diagnostic output off the operating system's root file system. + +Keep persistent data, the active WAL, the WAL archive, and node metadata on durable storage. +Do not place them in temporary directories such as `/tmp`, which may be cleaned automatically, or on volatile file systems such as `tmpfs`. +A temporary directory is not necessarily a `tmpfs` mount, but both are unsuitable for files that have to survive a host reboot. + +The Ignite-managed defaults are all relative to the work directory, so leaving the paths unset puts data, the active WAL, and the WAL archive on one file system. +By default, `storagePath` is `${IGNITE_HOME}/work/db/node{IDX}-{UUID}`, `walPath` is `${IGNITE_HOME}/work/db/wal/`, and `walArchivePath` is `${IGNITE_HOME}/work/db/wal/archive/`. +See link:persistence/native-persistence#configuring-persistent-storage-directory[Configuring Persistent Storage Directory]. + +== Configuration Example + +The example below places the main node paths on separate file systems: + +[source,java] +---- +include::{javaFile}[tags=layout-paths,indent=0] +---- + +The `/mnt/...` locations represent mount points prepared in advance; different directory names on one file system do not isolate anything. +The WAL archive and CDC directories share `/mnt/wal-archive`. + +See the following sections for the individual paths: + +* link:setup#configuring-work-directory[Configuring Work Directory] +* link:persistence/persistence-tuning#keep-wals-separately[Keep WALs Separately] +* link:persistence/snapshot-directory#configuring-snapshot-directory[Configuring Snapshot Directory] + +=== Additional Data and Snapshot Directories + +If the data is spread over several directories with `DataStorageConfiguration.extraStoragePaths`, configure the snapshot locations for them as well: +For full snapshots of data stored in `extraStoragePaths`, use `DataStorageConfiguration.extraSnapshotPaths` to set additional snapshot locations. +When configured, the two arrays must have the same length; entries at the same index correspond to each other. + +[source,java] +---- +include::{javaFile}[tags=extra-snapshot-paths,indent=0] +---- + +Additional snapshot paths are ordinary snapshot destinations: they do not have to be on the WAL archive file system. +Do not pass an explicit snapshot path together with them: when a path is specified for the snapshot itself, all snapshot files are stored inside it and the additional locations are not used. + +NOTE: The additional locations currently apply to full snapshots only. Support for extra storages in incremental snapshots is not implemented yet, see https://issues.apache.org/jira/browse/IGNITE-29128[IGNITE-29128^]. + +== Sizing the WAL Volume + +The active WAL volume has to hold the segments that exist at the same time plus a reserve for rotation. +A node keeps 10 active segments by default (`walSegments`), and they are filled in a cyclical order, so the volume has to hold that many segments of `walSegmentSize` plus the rotation reserve. +Changing `walSegmentSize` changes that footprint as well as the number of rotations. +Monitor the volume like the other volumes, and keep the reserve for the next segment: a write can fail inside an already allocated segment as well. + +The WAL mode defines the durability guarantees of a commit rather than the size of the volume, see link:persistence/native-persistence#wal-modes[WAL Modes]. + +Do not disable the WAL to stop unexpected volume growth: disabling or enabling it is only safe on a stable topology, and with the WAL disabled a node restart removes the data from its persistent storage. +See link:persistence/native-persistence#disabling-wal[Disabling WAL]. + +Archiving can be disabled by setting `walPath` to the same value as `walArchivePath`. +In that mode segments are written to the WAL directory under their absolute indexes instead of being copied into a separate directory, and the retained segments are removed according to `minWalArchiveSize` and `maxWalArchiveSize`. Review Comment: yep, that was written before this pr at this section: https://github.com/apache/ignite/blob/061f0bcaaad053a706ea58ea5c38cb434d325335/docs/_docs/persistence/native-persistence.adoc?plain=1#L318-L332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
