zstan commented on code in PR #13741: URL: https://github.com/apache/ignite/pull/13741#discussion_r4451700074
########## docs/_docs/server-node-file-system-layout.adoc: ########## @@ -0,0 +1,251 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + += Server Node File System Layout +:javaFile: {javaCodeDir}/PersistenceTuning.java + +This page brings together recommendations for placing server node files. +Use it to plan disk space, manage retained files, and protect node storage from log and diagnostic growth. + +These recommendations are intended for production server nodes. +On nodes without native persistence, the guidance for the work directory, logs, and JVM diagnostics still applies. + +== Separate File Systems + +Use separate file systems to keep log and diagnostic growth from taking space needed by the node. +For workloads with heavy writes, placing data files and the active WAL on separate devices can also improve write performance. +Choose a layout that fits your workload and available storage. +If several file types share a file system, plan for their combined use, limit retained logs and diagnostic files, and leave free space for checkpoints and WAL rotation. + +Running out of disk space can stop a node and prevent it from restarting. + +A separate file system bounds capacity: directories placed on the same file system share its free space. +It does not by itself reduce I/O contention, which requires a separate device or independently provisioned throughput. +Partitions of one physical disk isolate capacity but still share the failure risk of that disk. + +== Recommended Placement + +[cols="1,1,3", options="header"] +|=== +|Files |Configuration |Placement + +|Data and indexes +|`DataStorageConfiguration.storagePath` and `DataStorageConfiguration.extraStoragePaths` +|Dedicated, durable data file systems. Keep logs, diagnostic files, and retained backups off them, so that unrelated growth does not take the space needed by checkpoint writes and recovery. + +|Active WAL +|`DataStorageConfiguration.walPath` +|A separate file system with room for the active segments and a rotation reserve, see <<Sizing the WAL Volume>>. See <<Configuration Example>> for the path setting. On write-heavy workloads, a separate device for the active WAL and the data files reduces the competition between checkpoint page writes and WAL writes. + +|WAL archive +|`DataStorageConfiguration.walArchivePath` +|Preferably a separate file system. Plan its capacity for the retained segments, see <<Sizing the WAL Archive>>, and for the features that share it: CDC and incremental snapshots, see <<Account for Hard Links>>. + +|CDC +|`DataStorageConfiguration.cdcWalPath` +|The *same file system as the WAL archive*, because CDC uses hard links to archived WAL segments. A separate CDC directory does not isolate its disk-space usage. Bound its retention with `cdcWalDirectoryMaxSize`. + +|Work directory +|`IgniteConfiguration.workDirectory` +|Durable storage with free space to spare. The directory keeps binary metadata, marshaller mappings, and other node files even when data and WAL are configured elsewhere. Keep it apart from logs and diagnostic output, and do not treat it as disposable temporary storage. + +|Snapshots +|`IgniteConfiguration.snapshotPath` and `DataStorageConfiguration.extraSnapshotPaths` +|For full snapshots without increments, preferably a separate file system and, where available, separate storage: the snapshot adds a second read and write pass over the data. For incremental snapshots, the snapshot root must be on the same file system as the WAL archive, see <<Account for Hard Links>>. Keep independent backup copies outside the node storage. + +|Ignite logs +|Log4j2 appenders in the logging configuration; `java.util.logging.properties` and the `IGNITE_LOG_DIR` environment variable for the built-in JUL logger +|A separate file system, with a bound on the total retained volume rather than on individual files. When logs share a file system with data, their growth can take the space that checkpoints and WAL rotation need. `IGNITE_LOG_DIR` affects only the built-in JUL logger, not Log4j2 appenders. + +|JVM and OS diagnostics +|GC logging options; JFR repository and output settings; heap and core dump paths; monitoring-tool output +|Storage separate from data, WAL, and the work directory, with retained output bounded and space reserved for large dumps, see <<Logs and Diagnostic Output>>. +|=== + +If some roles have to share a file system, size it for their combined usage and limit what each role retains. +Keep Ignite files and diagnostic output off the operating system's root file system. + +Keep persistent data, the active WAL, the WAL archive, and node metadata on durable storage. +Do not place them in temporary directories such as `/tmp`, which may be cleaned automatically, or on volatile file systems such as `tmpfs`. +A temporary directory is not necessarily a `tmpfs` mount, but both are unsuitable for files that have to survive a host reboot. Review Comment: Seems you can olso mention that : logs and jfr can be stored in tmp/tmpfs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
