This is an automated email from the ASF dual-hosted git repository.
zstan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 8bd62d4eb54 IGNITE-29171 Document the recommended file system layout
for Ignite server nodes (#13741)
8bd62d4eb54 is described below
commit 8bd62d4eb54451001ab80a26c586f6e0946cfb69
Author: didar shayarov <[email protected]>
AuthorDate: Tue Nov 3 16:32:32 2026 +0300
IGNITE-29171 Document the recommended file system layout for Ignite server
nodes (#13741)
---
docs/_data/toc.yaml | 2 +
.../apache/ignite/snippets/PersistenceTuning.java | 44 ++++
docs/_docs/logging.adoc | 13 +-
.../persistence-tuning.adoc | 2 +
.../perf-and-troubleshooting/troubleshooting.adoc | 2 +
docs/_docs/persistence/change-data-capture.adoc | 4 +
docs/_docs/persistence/native-persistence.adoc | 49 +++-
docs/_docs/persistence/persistence-tuning.adoc | 2 +
docs/_docs/persistence/snapshot-directory.adoc | 6 +
docs/_docs/server-node-file-system-layout.adoc | 259 +++++++++++++++++++++
docs/_docs/setup.adoc | 11 +-
docs/_docs/snapshots/snapshots.adoc | 4 +
12 files changed, 387 insertions(+), 11 deletions(-)
diff --git a/docs/_data/toc.yaml b/docs/_data/toc.yaml
index cb50da29a88..8771e735699 100644
--- a/docs/_data/toc.yaml
+++ b/docs/_data/toc.yaml
@@ -59,6 +59,8 @@
url: understanding-configuration
- title: Setting Up
url: setup
+ - title: Server Node File System Layout
+ url: server-node-file-system-layout
- title: Configuring Logging
url: logging
- title: Resources Injection
diff --git
a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/PersistenceTuning.java
b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/PersistenceTuning.java
index 4371a95d868..dbbe3d93216 100644
---
a/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/PersistenceTuning.java
+++
b/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets/PersistenceTuning.java
@@ -106,4 +106,48 @@ public class PersistenceTuning {
ignite.close();
}
+ void fileSystemLayout() {
+ // tag::layout-paths[]
+ IgniteConfiguration cfg = new IgniteConfiguration();
+
+ // Work directory: node metadata, marshaller mappings, and other node
files.
+ cfg.setWorkDirectory("/mnt/node-meta/work");
+
+ // Destination of full snapshots.
+ cfg.setSnapshotPath("/mnt/snapshots");
+
+ DataStorageConfiguration storeCfg = new DataStorageConfiguration();
+
+ // Enabling native persistence for the default data region.
+
storeCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
+
+ // Data and indexes.
+ storeCfg.setStoragePath("/mnt/ignite-data/db");
+
+ // Active WAL segments.
+ storeCfg.setWalPath("/mnt/wal/db/wal");
+
+ // Retained WAL segments.
+ storeCfg.setWalArchivePath("/mnt/wal-archive/db/wal/archive");
+
+ // The CDC directory has to be on the same file system as the WAL
archive.
+ storeCfg.setCdcWalPath("/mnt/wal-archive/db/wal/cdc");
+
+ cfg.setDataStorageConfiguration(storeCfg);
+ // end::layout-paths[]
+ }
+
+ void extraSnapshotPaths() {
+ // tag::extra-snapshot-paths[]
+ DataStorageConfiguration storeCfg = new DataStorageConfiguration();
+
+ // Two data directories.
+ storeCfg.setExtraStoragePaths("/mnt/ignite-data-1/db",
"/mnt/ignite-data-2/db");
+
+ // A snapshot location for each data directory. The arrays must have
the same length,
+ // and the entries at the same index correspond to each other.
+ storeCfg.setExtraSnapshotPaths("/mnt/snapshots-1", "/mnt/snapshots-2");
+ // end::extra-snapshot-paths[]
+ }
+
}
diff --git a/docs/_docs/logging.adoc b/docs/_docs/logging.adoc
index 2f0fdccf976..164d3c0ccd6 100644
--- a/docs/_docs/logging.adoc
+++ b/docs/_docs/logging.adoc
@@ -35,7 +35,7 @@ When a node starts, it outputs start-up information to the
console, including th
| `IGNITE_LOG_INSTANCE_NAME` | If the property is set, Ignite includes its
instance name in log messages. | Not set
| `IGNITE_QUIET` | Set to `false` to disable the quiet mode and enable the
verbose mode.
In the verbose mode, the node logs a lot more information. | `true`
-| `IGNITE_LOG_DIR` | The directory where Ignite writes log files. |
`$IGNITE_HOME/
+| `IGNITE_LOG_DIR` | The directory where the built-in JUL logger writes log
files. It is read as an environment variable of the node process and has no
effect on Log4j2 appenders. | `$IGNITE_HOME/
work/log`
| `IGNITE_DUMP_THREADS_ON_FAILURE` | Set to `true` to output thread dumps to
the log when a critical error is caught. | `true`
|===
@@ -44,7 +44,11 @@ work/log`
== Default Logging
By default, Ignite uses the java.util.logging (JUL) framework.
If you start Ignite using the `ignite.sh|bat` script from the distribution
package, Ignite uses `$IGNITE_HOME/config/java.util.logging.properties` as the
default logging configuration file and outputs all messages to log files in the
`$IGNITE_HOME/work/log` directory.
-You can override the default logging directory by specifying the
`IGNITE_LOG_DIR` system property.
+You can override the default logging directory by setting the `IGNITE_LOG_DIR`
environment variable of the node process. This variable is read once at startup
and applies to the built-in JUL logger only; if you use Log4j2, change the file
paths in the Log4j2 configuration instead.
+
+NOTE: The default log directory is located inside the work directory, which
Ignite also uses to store metadata and, by default, data files.
+For production server nodes, bound total log retention and, where practical,
use a separate file system, so that log growth cannot exhaust the space
required by checkpoints, WAL, and node startup.
+See link:server-node-file-system-layout[Server Node File System Layout].
If you use Ignite as a library in your application, the default logging
configuration includes only console handler at INFO level.
You can provide a custom configuration file via the
`java.util.logging.config.file` system property.
@@ -76,6 +80,11 @@ tab:C++[unsupported]
In the above example, the path to `log4j2-config.xml` can be either an
absolute path, a local path relative to META-INF in classpath or to
`IGNITE_HOME`. An example log4j2 configuration file can be found in the
distribution package (`$IGNITE_HOME/config/ignite-log4j.xml`).
+NOTE: The configuration shipped in the distribution package writes to
`$IGNITE_HOME/work/log` and rolls individual files by time and by size.
+Its file name contains both an index and a date, so the implicit default
rollover strategy keeps a limited number of indexed archives of the current
period, but archives left from previous dates are never removed, and its
`consistency.log` appender is not rolled at all.
+Add a `Delete` action to `DefaultRolloverStrategy` to bound the directory as a
whole and bound `consistency.log` separately.
+See link:server-node-file-system-layout#logs-and-diagnostic-output[Server Node
File System Layout] for the recommended log placement and retention.
+
NOTE: Log4j2 supports runtime reconfiguration, i.e. changes in the
configuration file is applied without the need to restart the application.
NOTE: `log4j2-config.xml` enables colorization of console output by default.
This feature works out of the box on many operating systems, except for
Windows. If you want to enable this feature on Windows, please follow the
https://logging.apache.org/log4j/2.x/manual/layouts.html#ANSI_Styling_on_Windows[official
log4j2 instructions].
diff --git a/docs/_docs/perf-and-troubleshooting/persistence-tuning.adoc
b/docs/_docs/perf-and-troubleshooting/persistence-tuning.adoc
index 54cd57d63c8..bee1d429e1c 100644
--- a/docs/_docs/perf-and-troubleshooting/persistence-tuning.adoc
+++ b/docs/_docs/perf-and-troubleshooting/persistence-tuning.adoc
@@ -65,6 +65,8 @@ tab:C++[unsupported]
Consider using separate drives for data files and
link:persistence/native-persistence#write-ahead-log[Write-Ahead-Logging (WAL)].
Ignite actively writes to both the data and WAL files.
+Keeping the active WAL on its own device reduces the competition between
checkpoint page writes and WAL writes, which helps keep write latency stable
while a checkpoint is in progress.
+See link:server-node-file-system-layout[Server Node File System Layout] for
the recommended placement of the node files.
The example below shows how to configure separate paths for the data storage,
WAL, and WAL archive:
diff --git a/docs/_docs/perf-and-troubleshooting/troubleshooting.adoc
b/docs/_docs/perf-and-troubleshooting/troubleshooting.adoc
index 2ed39567109..6afe0151e10 100644
--- a/docs/_docs/perf-and-troubleshooting/troubleshooting.adoc
+++ b/docs/_docs/perf-and-troubleshooting/troubleshooting.adoc
@@ -29,6 +29,8 @@ On some systems, the default location for Ignite persistence
files might be unde
* Change the location of all persistence files using the
`DataStorageConfiguration` APIs, such as `setStoragePath(...)`,
`setWalPath(...)`, and `setWalArchivePath(...)`
+See link:server-node-file-system-layout[Server Node File System Layout] for
the recommended placement of persistence files, logs, and diagnostic output on
production server nodes.
+
== Cluster Doesn't Start After Field Type Changes
When developing your application, you may need to change the type of a custom
diff --git a/docs/_docs/persistence/change-data-capture.adoc
b/docs/_docs/persistence/change-data-capture.adoc
index 2f2464b0267..a83e1eda249 100644
--- a/docs/_docs/persistence/change-data-capture.adoc
+++ b/docs/_docs/persistence/change-data-capture.adoc
@@ -49,9 +49,13 @@ On startup event processing will be continued from the last
saved state.
|Name |Description | Default value
| `DataRegionConfiguration#cdcEnabled` | Flag to enable CDC on the server
node. | `false`
| `DataStorageConfiguration#cdcWalPath` | Path to the CDC directory |
`"db/wal/cdc"`
+| `DataStorageConfiguration#cdcWalDirectoryMaxSize` | Maximum size of the CDC
directory in bytes. If adding a segment would exceed the limit, the CDC link is
skipped and the changes in that segment are lost. | `0` (no limit)
| `DataStorageConfiguration#walForceArchiveTimeout` | Timeout to forcefully
archive the WAL segment even it is not complete. | `-1` (disabled)
|===
+NOTE: CDC links are hard links to WAL archive segments, so `cdcWalPath` must
be on the same file system as `walArchivePath`, and the space retained by CDC
is not bounded by `maxWalArchiveSize`.
+See link:server-node-file-system-layout#account-for-hard-links[Server Node
File System Layout] for the capacity planning rules.
+
=== CDC Application
CDC is configured in the same way as the Ignite node - via the spring XML file:
diff --git a/docs/_docs/persistence/native-persistence.adoc
b/docs/_docs/persistence/native-persistence.adoc
index 5ca6e86d623..6362fdd84f4 100644
--- a/docs/_docs/persistence/native-persistence.adoc
+++ b/docs/_docs/persistence/native-persistence.adoc
@@ -156,6 +156,10 @@ tab:C++[unsupported]
You can also change the WAL and WAL archive paths to point to directories
outside of the storage directory. Refer to the next section for details.
+NOTE: If the Ignite paths are left unset, data, the active WAL, the WAL
archive, and CDC all live in the work directory and therefore on one file
system.
+The CDC directory and the snapshot root of an incremental snapshot chain have
to be on the same file system as the WAL archive, because both use hard links
to archived segments.
+See link:server-node-file-system-layout[Server Node File System Layout] for
the recommended placement of the node directories, the capacity they need, and
how the write load affects the layout.
+
== Write-Ahead Log
The write-ahead log is a log of all data modifying operations (including
deletes) that happen on a node. When a page is updated in RAM, the update is
not directly written to the partition file but is appended to the tail of the
WAL.
@@ -207,11 +211,21 @@ If a node is terminated abruptly during update
operations, it is very likely tha
=== WAL Archive
-The WAL archive is used to store WAL segments that may be needed to recover
the node after a crash. The number of segments kept in the archive is such that
the total size of all segments does not exceed the specified size of the WAL
archive.
+The WAL archive is used to store WAL segments that may be needed to recover
the node after a crash. The number of segments kept in the archive is defined
by the cleanup rules described below.
+
+By default, the maximum size of the WAL archive is `maxWalArchiveSize`, whose
default value is 1 GiB. You can change that value in the <<Configuration
Properties,configuration>>.
+
+Cleanup starts when the archive exceeds `minWalArchiveSize`, not
`maxWalArchiveSize`. By default, `minWalArchiveSize` is half of
`maxWalArchiveSize`, that is 512 MiB.
+The cleanup runs asynchronously in the background. It removes the oldest
segments, stopping at the last completed checkpoint and at the segments that
are reserved by a running checkpoint or by a reader. It removes whole segments
while the archive stays above the threshold, so the remaining size can end up
below it.
+
+`maxWalArchiveSize` is the upper threshold, and it is not a strict disk-space
quota. Cleanup runs asynchronously and preserves the WAL needed for recovery,
so the archive can exceed this value. When the archive reaches it, Ignite
forcibly releases the reservations of the segments that the last completed
checkpoint already covers. This allows cleanup of older segments.
-By default, the maximum size of the WAL archive (total space it occupies on
disk) is defined as 4 times the size of the
link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpointing
buffer]. You can change that value in the <<Configuration
Properties,configuration>>.
+Removing an archived segment does not release its disk blocks while a CDC or
snapshot hard link still points to it, so the space taken on the file system
can be larger than the archive size.
-CAUTION: Setting the WAL archive size to a value lower than the default may
impact performance and should be tested before being used in production.
+CAUTION: Setting these thresholds to values lower than the defaults should be
tested before being used in production. The typical amount of retained history
follows `minWalArchiveSize`, so lowering it shortens the WAL history available
for recovery, incremental snapshots, and historical rebalancing.
+CDC is not affected: it keeps its own hard link to each captured segment, so
its history is retained independently of the archive cleanup and is bounded by
`cdcWalDirectoryMaxSize` instead.
+
+NOTE: `walHistorySize` is deprecated and ignored. The archive is controlled by
`maxWalArchiveSize` and `minWalArchiveSize` only.
:walXmlFile: code-snippets/xml/wal.xml
@@ -245,6 +259,8 @@ tab:C++[unsupported]
WARNING: Disabling or enabling WAL should only be done on stable topology: all
baseline nodes should be present, no nodes should join or leave the cluster for
the duration of this operation. Otherwise, cache may be stuck in inconsistent
state. If that happens, it is advised to destroy the affected cache(s).
There are situations when it is reasonable to have the WAL disabled to get
better performance. For instance, it is useful to disable WAL during initial
data loading and enable it after the pre-loading is complete.
+Disabling the WAL is a planned administrative operation with recovery
implications, not a way to reclaim disk space: follow the procedure in this
section and keep the files that the recovery needs.
+Do not remove WAL segments, checkpoint files, or other files of the persistent
store manually outside a documented maintenance or recovery procedure. Manual
changes to the internal persistent structure are not supported, and correct
cluster operation is not guaranteed after such an intervention.
////
todo: add c++ examples
@@ -275,13 +291,18 @@ ALTER TABLE Person LOGGING
tab:C++[unsupported]
--
-WARNING: If WAL is disabled and you restart a node, all data is removed from
the persistent storage on that node. This is implemented because without WAL
data consistency cannot be guaranteed in case of node crash or restart.
+WARNING: Disabling WAL through the runtime API removes local crash-recovery
guarantees for the affected cache groups.
+If a node stops while WAL is disabled, recovery may require maintenance and
cleanup of affected persistent files.
+`WALMode.NONE` has different shutdown and recovery behavior; see <<WAL Modes>>.
=== WAL Archive Compaction
+WAL archiving and WAL archive compaction are different mechanisms: archiving
copies completed segments to the archive directory without compressing them,
while compaction is a separate step that compresses segments that are already
archived.
+
You can enable WAL Archive compaction to reduce the space occupied by the WAL
Archive.
-By default, WAL Archive contains segments for the last 20 checkpoints (this
number is configurable).
+Compaction applies to the segments that are already archived; how much history
is kept is defined by `minWalArchiveSize` and `maxWalArchiveSize`, not by a
checkpoint count.
If compaction is enabled, all archived segments that are 1 checkpoint old are
compressed in ZIP format.
If the segments are needed (for example, to re-balance data between nodes),
they are uncompressed to RAW format.
+Compaction reduces the space the archive occupies, but it uses CPU and adds
I/O, so the actual cost depends on the load and on the compression level.
See the <<Configuration Properties>> section below to learn how to enable WAL
archive compaction.
@@ -329,6 +350,7 @@ It is safe to disable WAL archiving because a cluster
without the WAL archive pr
To disable archiving, set the WAL path and the WAL archive path to the same
value.
In this case, Ignite does not copy segments to the archive; instead, it
creates new segments in the WAL folder.
+WAL recording itself stays enabled: only the separate archive directory and
the copying of segments to it are disabled.
Old segments are deleted as the WAL grows, based on the WAL Archive size
setting.
@@ -342,7 +364,17 @@ Checkpointing and write-ahead logging are designed to
ensure durability of data
image:images/checkpointing-persistence.png[]
-This process helps to utilize disk space frugally by keeping pages in the most
up-to-date state on disk. After a checkpoint is passed, you can delete the WAL
segments that were created before that point in time.
+This process helps to utilize disk space frugally by keeping pages in the most
up-to-date state on disk. After a checkpoint completes, Ignite can clean up
older WAL segments when they are no longer needed.
+
+[[checkpoint-marker-retention]]
+=== Checkpoint Marker Retention
+
+Checkpoint progress is recorded in START and END marker files in the `cp`
directory of the node storage directory.
+With a finite `maxWalArchiveSize`, obsolete checkpoint marker files are
removed as the WAL cleanup boundary advances.
+If the WAL is written slowly, a large `walSegmentSize` can hold the records of
many checkpoints before the boundary moves to the next segment.
+Frequent checkpoints with changes can therefore leave many marker files in
`cp`.
+Consider `checkpointFrequency`, `walSegmentSize`, the WAL write rate, and the
archive retention together.
+When automatic WAL archive cleanup is disabled, Ignite attempts to trim
checkpoint history after each checkpoint according to the
`IGNITE_PDS_MAX_CHECKPOINT_MEMORY_HISTORY_SIZE` system property. WAL
reservations can delay this cleanup.
See the following related documentation:
@@ -368,5 +400,8 @@ The following table describes some properties of
link:{javadoc_base_url}/org/apa
|`walMode` | <<WAL Modes,Write-ahead logging mode>>. | `LOG_ONLY`
| `walCompactionLevel` | WAL archive compression level. `1` indicates the
fastest speed, and `9` indicates the best compression. | `1`
-|`maxWalArchiveSize` | The maximum size (in bytes) the WAL archive can occupy
on the file system. | Four times the size of the
link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpointing
buffer].
+|`maxWalArchiveSize` | The upper threshold (in bytes) of the WAL archive.
When the archive reaches it, Ignite forcibly releases the reservations of the
segments that the last completed checkpoint already covers, so that the cleanup
can delete them. It is not a strict disk-space quota: the cleanup runs
asynchronously and preserves the WAL needed for recovery, so the archive can
exceed this value. | 1 GiB
+|`minWalArchiveSize` | The threshold (in bytes) that makes segments eligible
for cleanup. The cleanup runs in the background and removes the oldest segments
while the archive stays above this value. Segments are removed as whole files,
so the remaining size can be below it. | Half of `maxWalArchiveSize`
+|`walSegments` | The number of fixed-size segments used for active WAL
rotation when archiving is enabled. Plan space for `walSegments` ×
`walSegmentSize` bytes plus a rotation reserve. Size the WAL archive
separately. | `10`
+|`checkpointFrequency` | The interval between scheduled checkpoints, in
milliseconds. | `180000` (3 minutes)
|=======================================================================
diff --git a/docs/_docs/persistence/persistence-tuning.adoc
b/docs/_docs/persistence/persistence-tuning.adoc
index 293d9de110e..e581ad7f67e 100644
--- a/docs/_docs/persistence/persistence-tuning.adoc
+++ b/docs/_docs/persistence/persistence-tuning.adoc
@@ -63,6 +63,8 @@ tab:C++[unsupported]
Consider using separate drives for data files and
link:persistence/native-persistence#write-ahead-log[Write-Ahead-Logging (WAL)].
Ignite actively writes to both the data and WAL files.
+Keeping the active WAL on its own device reduces the competition between
checkpoint page writes and WAL writes, which helps keep write latency stable
while a checkpoint is in progress.
+See link:server-node-file-system-layout[Server Node File System Layout] for
the recommended placement of the node files.
The example below shows how to configure separate paths for the data storage,
WAL, and WAL archive:
diff --git a/docs/_docs/persistence/snapshot-directory.adoc
b/docs/_docs/persistence/snapshot-directory.adoc
index a046fabb31a..10143b702a9 100644
--- a/docs/_docs/persistence/snapshot-directory.adoc
+++ b/docs/_docs/persistence/snapshot-directory.adoc
@@ -24,6 +24,12 @@ Ignite Persistence routines, it's suggested to store the
snapshot and persistenc
You can avoid this interference between Ignite Native persistence and
snapshotting
by either changing
link:persistence/native-persistence#configuring-persistent-storage-directory[storage
directories of the persistence files]
or overriding the default snapshots' location as shown below:
+
+NOTE: For full snapshots without increments, prefer a separate file system
where the available storage allows it, and separate storage where available:
the snapshot adds a second read and write pass over the data and competes with
the traffic the cluster is serving.
+If you create incremental snapshots, the snapshot root of the base full
snapshot and of its increments has to be on the *same file system as the WAL
archive*, because increments are hard links to archived WAL segments.
+Different partitions of one physical disk do not satisfy this requirement, and
the retained chains occupy space after the archive cleanup.
+See link:server-node-file-system-layout[Server Node File System Layout] for
the complete placement and capacity plan.
+
[tabs]
--
tab:XML[]
diff --git a/docs/_docs/server-node-file-system-layout.adoc
b/docs/_docs/server-node-file-system-layout.adoc
new file mode 100644
index 00000000000..9d7c83e48d6
--- /dev/null
+++ b/docs/_docs/server-node-file-system-layout.adoc
@@ -0,0 +1,259 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Server Node File System Layout
+:javaFile: {javaCodeDir}/PersistenceTuning.java
+
+This page brings together recommendations for placing the directories used by
the Ignite server node: data and indexes, the active WAL, the WAL archive, CDC,
the work directory, snapshots, logs, and JVM diagnostics.
+Use it to plan disk space, manage retained files, and keep log and diagnostic
growth away from the space that the node needs.
+
+These recommendations are intended for production server nodes.
+On nodes without native persistence, the guidance for the work directory,
logs, and JVM diagnostics still applies.
+
+== Separate File Systems
+
+Node directories differ in how they grow and how they are written.
+For write-heavy workloads, place the data files and the active WAL on separate
devices: they are written in different patterns and otherwise compete for the
same device.
+Other node directories can share a file system. If they do, size the file
system for their combined volume, bound the retained logs and diagnostic
output, and leave free space for checkpoints and WAL rotation.
+A dedicated file system keeps unrelated growth away from the space the node
needs, and it separates the node from the other software on the host.
+
+Running out of disk space can stop a node and prevent it from restarting.
+
+Start with the split between the data files and the active WAL, and add
volumes for the other node directories only where their growth, retention, or
I/O patterns require it.
+The <<Recommended Placement>> section lists the preferred placement for each
node directory and the capacity it needs.
+
+== Recommended Placement
+
+[cols="1,1,3", options="header"]
+|===
+|Node directories |Configuration |Placement
+
+|Data and indexes
+|`DataStorageConfiguration.storagePath` and
`DataStorageConfiguration.extraStoragePaths`
+|Durable data file systems. On write-heavy workloads, give them a device of
their own, see <<Separate File Systems>>. Keep logs, diagnostic files, and
retained backups off them, so that unrelated growth does not take the space
needed by checkpoint writes and recovery.
+
+|Active WAL
+|`DataStorageConfiguration.walPath`
+|Its own file system, and on write-heavy workloads its own device, with room
for the active segments and a rotation reserve, see <<Sizing the WAL Volume>>.
See <<Configuration Example>> for the path setting. A separate device for the
active WAL and the data files reduces the competition between checkpoint page
writes and WAL writes.
+
+|WAL archive
+|`DataStorageConfiguration.walArchivePath`
+|A separate file system where the available storage allows it, because the
archive grows independently of the data volume. Otherwise plan its capacity
together with the other node directories that share the file system. Size it
for the retained segments, see <<Sizing the WAL Archive>>, and for the features
that share it: CDC and incremental snapshots, see <<Account for Hard Links>>.
+
+|CDC
+|`DataStorageConfiguration.cdcWalPath`
+|The *same file system as the WAL archive*, because CDC uses hard links to
archived WAL segments. A separate CDC directory does not isolate its disk-space
usage. Bound its retention with `cdcWalDirectoryMaxSize`.
+
+|Work directory
+|`IgniteConfiguration.workDirectory`
+|Durable storage with free space to spare. The directory keeps binary
metadata, marshaller mappings, and other node files even when data and WAL are
configured elsewhere. Do not treat it as disposable temporary storage, and keep
logs and diagnostic output out of the space it needs.
+
+|Snapshots
+|`IgniteConfiguration.snapshotPath` and
`DataStorageConfiguration.extraSnapshotPaths`
+|For full snapshots without increments, a separate file system where the
available storage allows it, and, where available, separate storage: the
snapshot adds a second read and write pass over the data. For incremental
snapshots, the snapshot root must be on the same file system as the WAL
archive, see <<Account for Hard Links>>. Keep independent backup copies outside
the node storage.
+
+|Ignite logs
+|Log4j2 appenders in the logging configuration; `java.util.logging.properties`
and the `IGNITE_LOG_DIR` environment variable for the built-in JUL logger
+|Storage where the logs cannot take the space that data, WAL, and checkpoints
need: a separate file system where the available storage allows it, otherwise a
bounded retained volume. Bound the total retained volume rather than individual
files. `IGNITE_LOG_DIR` affects only the built-in JUL logger, not Log4j2
appenders.
+
+|JVM and OS diagnostics
+|GC logging options; JFR repository and output settings; heap and core dump
paths; monitoring-tool output
+|Storage that is not needed by data, WAL, and the work directory, with
retained output bounded and space reserved for large dumps, see <<Logs and
Diagnostic Output>>.
+|===
+
+Keep Ignite files and diagnostic output off the operating system's root file
system.
+
+Keep persistent data, the active WAL, the WAL archive, and node metadata on
durable storage.
+Do not place them in temporary directories such as `/tmp`, which may be
cleaned automatically, or on volatile file systems such as `tmpfs`.
+A temporary directory is not necessarily a `tmpfs` mount, but both are
unsuitable for files that have to survive a host reboot.
+
+Logs and JFR data may use temporary storage if losing them on cleanup or a
host reboot is acceptable.
+Limit retained output, and account for memory use when using `tmpfs`.
+See <<Logs and Diagnostic Output>> for the JFR repository requirements.
+
+The Ignite-managed defaults are all relative to the work directory, so leaving
the paths unset puts data, the active WAL, and the WAL archive on one file
system.
+By default, `storagePath` is `${IGNITE_HOME}/work/db/node{IDX}-{UUID}`,
`walPath` is `${IGNITE_HOME}/work/db/wal/`, and `walArchivePath` is
`${IGNITE_HOME}/work/db/wal/archive/`.
+See
link:persistence/native-persistence#configuring-persistent-storage-directory[Configuring
Persistent Storage Directory].
+
+If several nodes share a host, give each node its own work directory and its
own set of node directories, and plan the file systems for their combined use.
+See link:perf-and-troubleshooting/general-perf-tips[General Performance Tips]
for the recommended number of server nodes per host.
+
+== Configuration Example
+
+The example below shows one possible layout.
+
+[source,java]
+----
+include::{javaFile}[tags=layout-paths,indent=0]
+----
+
+The `/mnt/...` locations represent mount points prepared in advance; different
directory names on one file system do not isolate anything.
+The WAL archive and CDC directories share `/mnt/wal-archive`.
+
+See the following sections for the individual paths:
+
+* link:setup#configuring-work-directory[Configuring Work Directory]
+* link:persistence/persistence-tuning#keep-wals-separately[Keep WALs
Separately]
+*
link:persistence/snapshot-directory#configuring-snapshot-directory[Configuring
Snapshot Directory]
+
+=== Additional Data and Snapshot Directories
+
+If the data is spread over several directories with
`DataStorageConfiguration.extraStoragePaths`, configure the snapshot locations
for them as well:
+For full snapshots of data stored in `extraStoragePaths`, use
`DataStorageConfiguration.extraSnapshotPaths` to set additional snapshot
locations.
+When configured, the two arrays must have the same length; entries at the same
index correspond to each other.
+
+[source,java]
+----
+include::{javaFile}[tags=extra-snapshot-paths,indent=0]
+----
+
+Additional snapshot paths are ordinary snapshot destinations: they do not have
to be on the WAL archive file system.
+Do not pass an explicit snapshot path together with them: when a path is
specified for the snapshot itself, all snapshot files are stored inside it and
the additional locations are not used.
+
+NOTE: Additional snapshot locations currently apply to full snapshots only.
They are not supported for incremental snapshots.
+
+== Sizing the WAL Volume
+
+The active WAL volume has to hold the segments that exist at the same time
plus a reserve for rotation.
+A node keeps 10 active segments by default (`walSegments`), and they are
filled in a cyclical order, so the volume has to hold that many segments of
`walSegmentSize` plus the rotation reserve.
+Changing `walSegmentSize` changes that footprint as well as the number of
rotations.
+Large segments combined with frequent checkpoints and a low WAL write rate can
increase the number of retained checkpoint marker files, see
link:persistence/native-persistence#checkpoint-marker-retention[Checkpoint
Marker Retention].
+Monitor the volume like the other volumes, and keep the reserve for the next
segment: a write can fail inside an already allocated segment as well.
+
+The WAL mode defines the durability guarantees of a commit rather than the
size of the volume, see link:persistence/native-persistence#wal-modes[WAL
Modes].
+
+Do not disable the WAL to stop unexpected volume growth.
+Disabling the WAL is a planned operation, such as initial data loading,
performed on a stable topology, and it changes what the node can recover:
follow the documented procedure and keep the files that the recovery needs.
+Do not remove WAL segments, checkpoint files, or other files of the persistent
store manually outside a documented maintenance or recovery procedure, because
correct cluster operation is not guaranteed after such an intervention.
+See link:persistence/native-persistence#disabling-wal[Disabling WAL].
+
+Setting `walPath` and `walArchivePath` to the same directory disables copying
segments to a separate archive directory, not WAL recording.
+In that mode segments are written to the WAL directory under their absolute
indexes instead of being copied into a separate directory, and the retained
segments are removed according to `minWalArchiveSize` and `maxWalArchiveSize`.
+Plan the capacity of that file system like the capacity of a data volume,
because it holds both the active and the retained segments.
+Incremental snapshots require the WAL archive to be enabled, and CDC receives
its links from the archiver, so standalone CDC does not capture new changes
without an archive.
+See link:persistence/native-persistence#disabling-wal-archive[Disabling WAL
Archive].
+
+== Sizing the WAL Archive
+
+Cleanup starts when the archive reaches `minWalArchiveSize`. By default, this
threshold is half of `maxWalArchiveSize`: 512 MiB with the default maximum of 1
GiB.
+See link:persistence/native-persistence#wal-archive[WAL Archive] for the
cleanup rules.
+
+The cleanup runs in the background and deletes whole segments, so the
remaining size can end up below the threshold.
+It does not remove the segments that the recovery still needs.
+`maxWalArchiveSize` is not a strict disk-space quota: cleanup runs
asynchronously and preserves the WAL that the recovery needs, so the archive
can exceed this value.
+When the archive reaches `maxWalArchiveSize`, Ignite releases the reservations
that the last completed checkpoint already covers, which allows cleanup of
older segments.
+
+Removing an archived segment does not release its disk blocks while a CDC or
snapshot hard link still points to it, so plan the capacity of the file system
from the retained links as well, see <<Account for Hard Links>>.
+
+Size the file system for the archive retention, the expected CDC backlog, and
the retained incremental snapshot chains.
+Measure the retained WAL volume under representative load, account for the
compression ratio when WAL archive compaction is enabled, and leave free space
for temporary growth.
+
+Lowering the thresholds below their defaults is allowed but should be tested:
the typical amount of retained history follows `minWalArchiveSize`, so lowering
it shortens the WAL history available for recovery, incremental snapshots, and
historical rebalancing.
+CDC is not affected: it keeps its own hard link to each captured segment, so
its history is bounded by `cdcWalDirectoryMaxSize` instead.
+Retention is not controlled by `walHistorySize`: the parameter is deprecated
and ignored.
+
+WAL archiving copies completed segments to the archive directory; it does not
compress them.
+WAL archive compaction is a separate option, disabled by default, that
compresses eligible segments into ZIP files: it reduces disk space but uses CPU
and adds I/O.
+See link:persistence/native-persistence#wal-archive-compaction[WAL Archive
Compaction].
+
+Capacity is not the only requirement: the archiver has to write at the peak
rate at which the active WAL produces segments, and the same device may also
serve compaction and the consumers that read the archive.
+If the archiver falls behind, the working segments run out and the write waits
for space that only archiving can free.
+Provision sequential throughput for the peak WAL rate plus compaction, and
verify it on the archive device itself.
+
+Archived segments are written once, mostly sequentially, and are read again
only for recovery, CDC, incremental snapshots, historical rebalancing, and
archive compaction.
+This makes the archive a candidate for cheaper, higher-capacity, slower media,
such as HDDs, while fast devices stay dedicated to the data and the active WAL.
+Slower media also extend the time to replay archived segments during recovery,
to catch up CDC, and to create an incremental snapshot, so take these
objectives into account when choosing the device.
+
+== Account for Hard Links
+
+Keep the CDC directory and the snapshot root used for incremental snapshots on
the same file system as the WAL archive.
+Both features use hard links to archived WAL segments.
+
+The links are created on segment files, not on the CDC or snapshot
directories, so a separate directory on the same file system does not separate
its free space.
+
+For CDC, `maxWalArchiveSize` does not bound the retained space.
+Configure a positive `cdcWalDirectoryMaxSize` appropriate for the expected lag
and for the capacity shared with the archive; the default is unlimited.
+This setting does not reserve free space, so leave room for the archive and
the other users of the file system.
+If adding a segment would exceed the limit, Ignite skips its CDC link and the
corresponding changes are unavailable to CDC.
+The skip is reported in the node log, and `ignite-cdc.sh` notices the gap only
when it reaches it, which can happen long after the event.
+Treat the first skipped segment as a reason to re-sync CDC, because this limit
trades CDC continuity for bounded retention.
+
+For incremental snapshots, the snapshot root of the base full snapshot and of
its increments has to be on the same file system as the WAL archive.
+Snapshot links can retain disk blocks after the archive cleanup, so include
the retained snapshot chains in the capacity plan.
+Additional snapshot locations configured with `extraSnapshotPaths` do not
change this requirement and are not used for incremental snapshots.
+
+Even when a full snapshot destination is on a separate file system,
concurrent-change tracking can require temporary space on the live data file
systems, so reserve space on both the source and the destination.
+If the destination has its own file system, a destination that fills stops the
snapshot operation while the node keeps serving; if the snapshot shares a file
system with data or WAL, the same exhaustion can stop the node.
+Store independent backup copies outside the node storage.
+
+== Logs and Diagnostic Output
+
+Ignite does not bound the log directory as a whole.
+Each logger bounds something slightly different, and the defaults are weaker
than they look, so check the effective configuration instead of assuming that
rotation is enough.
+
+* *Log4j2.* The default `config/ignite-log4j.xml` writes to
`${IGNITE_HOME}/work/log` and rolls files by time and by size (10 MB).
+Its file name contains both an index and a date, so the implicit default
rollover strategy keeps a limited number of indexed archives of the current
period, but nothing removes the archives left from previous dates.
+Add a `Delete` action to `DefaultRolloverStrategy`, by age or by total size,
so that the directory as a whole stays bounded.
+The `consistency.log` appender is a plain file appender and is not rolled at
all; bound it separately.
+* *JUL.* `config/java.util.logging.properties` sets both `limit` and `count`.
+The file name contains the node ID, so `count` bounds the file set of one node
ID only, and the sets produced by earlier node IDs stay on disk.
+Clean them up separately or size the volume for several generations.
+The `IGNITE_LOG_DIR` environment variable relocates this directory; it has no
effect on the Log4j2 appenders.
+* *GC logs.* For JVM unified logging, rotation parameters belong to the
output-options section and go after the decorator list, for example
`-Xlog:gc*:file=<path>::filecount=N,filesize=M`.
+* *stdout and stderr.* Ignite does not rotate a redirected stdout file.
+Prefer capturing and rotating the output with the service manager or the
container runtime.
+If you redirect it to a file yourself, rotate that file externally and be
aware of two limits: deleting the file does not release its space while the JVM
keeps it open, and `copytruncate` needs space for the copy and can lose the
lines written between the copy and the truncation.
+Enable console appenders only if you intend to capture their output, otherwise
redirecting stdout duplicates the log.
+* *Diagnostics.* Heap dumps, core dumps, saved recordings, and monitoring
output are not covered by any of the above and must be managed separately.
+
+Size the log volume for the total retention that the configured policies can
leave behind, plus the captured stdout.
+
+For JFR, configure both the disk repository and the recording output locations.
+By default the repository is created in the temporary directory of the
operating system, and setting a recording filename alone does not relocate it.
+`maxsize` and `maxage` are per-recording options: they bound the data that
each individual recording keeps in the repository, not the size of the
repository as a whole and not the age of the `.jfr` files already written out.
+Some JDK versions can terminate the JVM if the JFR repository runs out of disk
space. This bypasses the Ignite failure handler, so a separate diagnostic file
system protects the Ignite data volumes from JFR growth but does not prevent
this JVM failure. Set retention limits for each recording, reserve free space
for concurrent recordings and dumps, and monitor repository usage.
+
+The Control Script writes through the same logger configuration and opens its
log file before the command runs, so a full or unwritable log directory can
prevent the utility from starting.
+Run it from a host where the log directory is writable and has free space.
+
+== Plan Free Space
+
+Size every backing file system with free space to spare, including those used
by additional data paths and by diagnostic tools.
+Monitor the space that the logs and diagnostic files take.
+On Linux, monitor both available bytes and free inodes, taking account of
quotas and the space available to the Ignite operating-system user.
+
+Free space is not the only property of storage that matters.
+Check storage performance under representative load: use checkpoint duration,
page-write time, and fsync time to assess whether the storage keeps up with the
updates, see link:monitoring-metrics/new-metrics#data-storage[Data Storage].
+For performance checks, monitor device IOPS, throughput, and I/O latency
alongside these metrics and the WAL write rate (`WalWritingRate`).
+
+There is no universal safe reserve.
+It depends on the write rate, the size of the data region, the checkpoint
interval, and the time needed to react, so derive it from measurements on your
own workload: the growth of the file system under the peak expected load and
the time needed to react and free space.
+Leave enough free space for that growth and for checkpoints and WAL rotation
to finish.
+
+Stopping application writes does not stop disk growth: the checkpoint that
follows still has to flush dirty pages.
+That is why the reaction belongs at a threshold rather than at zero free space.
+
+The reserve also has a hardware side: random writes on a half-filled SSD are
substantially faster than on a nearly full one, because of over-provisioning.
+Choosing devices with a higher over-provisioning rate, and leaving room to use
it, is part of the same decision.
+See link:persistence/persistence-tuning#ssd-over-provisioning[SSD
Over-provisioning].
+
+== Related Configuration
+
+* link:setup#configuring-work-directory[Configuring Work Directory]
+* link:persistence/persistence-tuning#keep-wals-separately[Keep WALs
Separately]
+*
link:persistence/snapshot-directory#configuring-snapshot-directory[Configuring
Snapshot Directory]
+* link:persistence/native-persistence[Ignite Persistence]
+* link:persistence/change-data-capture[Change Data Capture]
+* link:snapshots/snapshots[Cluster Snapshots]
+* link:logging[Configuring Logging]
diff --git a/docs/_docs/setup.adoc b/docs/_docs/setup.adoc
index 0c758942c6a..79e234180f9 100644
--- a/docs/_docs/setup.adoc
+++ b/docs/_docs/setup.adoc
@@ -146,6 +146,10 @@
include::code-snippets/cpp/src/setting_work_directory.cpp[tag=setting-work-direc
----
--
+The work directory contains binary metadata and marshaller mappings, so treat
it as durable storage even when data and WAL are configured elsewhere.
+Keep the work directory on persistent storage with enough free space.
+See link:server-node-file-system-layout[Server Node File System Layout] for
the recommended disk placement of the work directory, data, WAL, logs, and
diagnostic files.
+
== Enabling Modules
@@ -345,6 +349,8 @@ If you are going to use either binary distribution or
Maven, you are
encouraged to set up the work directory for Ignite.
The work directory is used to store metadata information, index files, your
application data (if you use the link:persistence/native-persistence[Native
Persistence] feature), logs, and other files.
We recommend you always set up the work directory.
+When data, WAL, and logs are configured to separate locations, keep the work
directory on persistent storage with enough free space.
+See link:server-node-file-system-layout[Server Node File System Layout].
=== Recommended Logging Configuration
@@ -354,6 +360,7 @@ Logs play an important role when it comes to
troubleshooting and finding what we
* Start Ignite in verbose mode:
- If you use `ignite.sh`, specify the `-v` option.
- If you start Ignite from Java code, set the following system variable:
`IGNITE_QUIET=false`.
-* Do not store log files in the `/tmp` folder. This folder is cleared up every
time the server is restarted.
-* Make sure that there is enough space available on the storage where the log
files are stored.
+* Use durable storage for logs that you need after a host reboot. Temporary
storage is suitable only for logs that can be discarded.
+* Make sure that there is enough space available on the storage where the log
files are stored, and bound the total retained volume, not only the size of
individual files.
* Archive old log files periodically to save on storage space.
+* Keep log growth away from the space that data, WAL, and node metadata need.
See link:server-node-file-system-layout[Server Node File System Layout].
diff --git a/docs/_docs/snapshots/snapshots.adoc
b/docs/_docs/snapshots/snapshots.adoc
index 8a18cc5cbdb..78cc0caa84a 100644
--- a/docs/_docs/snapshots/snapshots.adoc
+++ b/docs/_docs/snapshots/snapshots.adoc
@@ -115,6 +115,10 @@ Ignite Persistence routines, it's suggested to store the
snapshot and persistenc
See the
link:persistence/snapshot-directory#configuring-snapshot-directory[Configuring
Snapshot Directory] page for
configuration examples.
+NOTE: A full snapshot reads all the data and writes a separate stream of
files, so on a shared device it adds a second full read and write pass on top
of the traffic the cluster is serving.
+The snapshot root used for incremental snapshots has to be on the same file
system as the WAL archive, because increments are hard links to archived WAL
segments.
+See link:server-node-file-system-layout[Server Node File System Layout] for
capacity and placement recommendations.
+
=== Snapshot Execution Pool
By default, the snapshot thread pool size has a value of `4`. Decreasing the
number of threads involved in the snapshot creation process