[
https://issues.apache.org/jira/browse/CASSANDRA-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382124#comment-16382124
]
Kenneth Brotman edited comment on CASSANDRA-14272 at 3/1/18 4:05 PM:
---------------------------------------------------------------------
>From DataStax at
>[https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupRestoreTOC.html]
was (Author: kenbrotman):
>From DataStax at
>[https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupRestoreTOC.html]
h1. Backing up and restoring data
DataStax OpsCenter provides automated backup and restore functionality, see
[Backup
Service|https://docs.datastax.com/en/opscenter/latest/opsc/online_help/services/opscBackupService.html].
* *[About
snapshots|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsAboutSnapshots.html]*
A brief description of how DataStax Enterprise backs up data.
* *[Taking a
snapshot|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupTakesSnapshot.html]*
Steps for taking a global snapshot or per node.
* *[Deleting snapshot
files|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupDeleteSnapshot.html]*
Steps to delete snapshot files.
* *[Enabling incremental
backups|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupIncremental.html]*
Steps to enable incremental backups. When incremental backups are enabled,
DataStax Enterprise hard-links each memtable flushed to an SSTable to a backups
directory under the keyspace data directory.
* *[Restoring from a
snapshot|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupSnapshotRestore.html]*
Methods for restoring from a snapshot.
* *[Restoring a snapshot into a new
cluster|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsSnapshotRestoreNewCluster.html]*
Steps for restoring a snapshot by recovering the cluster into another newly
created cluster.
* *[Recovering from a single disk failure using
JBOD|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html]*
Steps for recovering from a single disk failure in a disk array using JBOD
(just a bunch of disks).
h1. About snapshots
DataStax Enterprise backs up data by taking a snapshot of all on-disk data
files (SSTable files) stored in the data directory. You can take a snapshot of
all keyspaces, a single keyspace, or a single table while the system is online.
Using a parallel ssh tool (such as pssh), you can snapshot an entire cluster.
This provides an eventually consistent backup. Although no one node is
guaranteed to be consistent with its replica nodes at the time a snapshot is
taken, a restored snapshot resumes consistency using built-in consistency
mechanisms.
After a system-wide snapshot is performed, you can enable incremental backups
on each node to backup data that has changed since the last snapshot. Each time
a memtable is flushed to disk and an SSTable is created, a hard link is copied
into a {color:#374c51}/backups{color} subdirectory of the data directory
(provided JNA is enabled). Compacted SSTables do not create hard links in
{color:#374c51}/backups{color} because these SSTables do not contain any data
that has not already been linked.
h1. Taking a snapshot
Snapshots are taken per node using the [nodetool
snapshot|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsSnapShot.html]
command. To take a global snapshot, run the {{nodetool snapshot}} command with
a parallel ssh utility, such as pssh.
A snapshot first
[flushes|https://docs.datastax.com/en/dse/5.1/dse-arch/datastax_enterprise/dbInternals/dbIntHowDataMaintain.html]
all in-memory writes to disk, then makes a hard link of the SSTable files for
each keyspace. You must have enough free disk space on the node to accommodate
making snapshots of your data files. A single snapshot requires little disk
space. However, snapshots can cause your disk usage to grow more quickly over
time because a snapshot prevents old obsolete data files from being deleted.
After the snapshot is complete, you can move the backup files to another
location if needed, or you can leave them in place.
Note: Restoring from a snapshot requires the table schema.
h2. Procedure
Run the nodetool snapshot command, specifying the hostname, JMX port, and
keyspace. For example:
{color:#990073}$ {color}nodetool snapshot -t cycling_2017-3-9 cycling
Tarball and Installer No-Services path:
{{installation_location{color:#009926}/resources/{color}cassandra/bin}}
h2. Results
The name of the snapshot directory appears:{{{color:#000080}Requested{color}
{color:#000080}creating{color}
{color:#000080}snapshot{color}({color:#000080}s{color})
{color:#000080}for{color} [cycling] {color:#000080}with{color}
{color:#000080}snapshot{color} {color:#000080}name{color} [2015.07.17]
{color:#000080}Snapshot{color} {color:#000080}directory{color}:
{color:#000080}cycling_2017-3-9{color}}}
The snapshot files are created in
{color:#374c51}data/keyspace_name/table_name-UUID/snapshots/snapshot_name{color}
directory.
ls -1
data{color:#009926}/cycling/{color}cyclist_name-9e516080f30811e689e40725f37c761d{color:#009926}/snapshots/{color}cycling_2017-3-9
For all installations, the default location of the {color:#374c51}data{color}
directory is {color:#374c51}/var/lib/cassandra/data{color}.
The data files extension is {color:#374c51}.db{color} and the full CQL to
create the table is in the {color:#374c51}schema.cql{color}
file.{{manifest.json mc-1-big-CompressionInfo.db mc-1-big-Data.db
mc-1-big-Digest.crc32 mc-1-big-Filter.db mc-1-big-Index.db
mc-1-big-Statistics.db mc-1-big-Summary.db mc-1-big-TOC.txt schema.cql}}
h1. Deleting snapshot files
When taking a snapshot, previous snapshot files are not automatically deleted.
You should remove old snapshots that are no longer needed.
The [nodetool
clearsnapshot|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsClearSnapShot.html]
command removes all existing snapshot files from the snapshot directory of
each keyspace. You should make it part of your back-up process to clear old
snapshots before taking a new one.
h2. Procedure
# To delete all snapshots for a node, run the nodetool
{color:#374c51}clearsnapshot{color} command. For example:
nodetool -h localhost -p 7199 clearsnapshot
Tarball and Installer No-Services path:
{{installation_location{color:#009926}/resources/{color}cassandra/bin}}
To delete snapshots on all nodes at once, run the nodetool clearsnapshot
command using a parallel ssh utility.
# To delete a single snapshot, run the {color:#374c51}clearsnapshot{color}
command with the snapshot name:
nodetool clearsnapshot -t <snapshot_name>
The file name and path vary according to the type of snapshot. See [nodetools
snapshot
|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsSnapShot.html]
for details about snapshot names and paths.
h1. Enabling incremental backups
When incremental backups are enabled (disabled by default), DataStax Enterprise
hard-links each memtable-flushed SSTable to a backups directory under the
keyspace data directory. This allows storing backups offsite without
transferring entire snapshots. Also, incremental backups combined with
snapshots to provide a dependable, up-to-date backup mechanism. Compacted
SSTables do not create hard links in {color:#374c51}/backups{color} because
these SSTables do not contain any data that has not already been linked.A
snapshot at a point in time, plus all incremental backups and commit logs since
that time form a compete backup.
As with snapshots, DataStax Enterprise does not automatically clear incremental
backup files. DataStax recommends setting up a process to clear incremental
backup hard-links each time a new snapshot is created.
h2. Procedure
Edit the
[cassandra.yaml|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupIncremental.html#opsBackupIncremental__cassandrayaml]
configuration file on each node in the cluster and change the value of
[incremental_backups|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configCassandra_yaml.html#configCassandra_yaml__incremental_backups]
to true.
h1. Restoring from a snapshot
Methods for restoring from a snapshot.
Restoring a keyspace from a snapshot requires all snapshot files for the table,
and if using incremental backups, any incremental backup files created after
the snapshot was taken. Streamed SSTables (from repair, decommission, and so
on) are also hardlinked and included.
Note: Restoring from snapshots and incremental backups temporarily causes
intensive CPU and I/O activity on the node being restored.
h2. Restoring from local nodes
This method copies the SSTables from the snapshots directory into the correct
data directories.
# Make sure the table schema exists and is the same as when the snapshot was
created.The nodetool snapshot command creates a table schema in the output
directory. If the table does not exist, recreate it using the schema.cql file.
# If necessary,
[{color:#0066cc}truncate{color}|https://docs.datastax.com/en/dse/5.1/cql/cql/cql_reference/cql_commands/cqlTruncate.html]
the table.
Note: You may not need to truncate under certain conditions. For example, if a
node lost a disk, you might restart before restoring so that the node continues
to receive new writes before starting the restore procedure.
Truncating is usually necessary. For example, if there was an accidental
deletion of data, the tombstone from that delete has a later write timestamp
than the data in the snapshot. If you restore without truncating (removing the
tombstone), the database continues to shadow the restored data. This behavior
also occurs for other types of overwrites and causes the same problem.
# Locate the most recent snapshot folder. For
example:/var/lib/cassandra/data/keyspace_name/table_name-UUID/snapshots/snapshot_name
# Copy the most recent snapshot SSTable directory to the
/var/lib/cassandra/data/keyspace/table_name-UUID directory.
For all installations, the default location of the data directory is
/var/lib/cassandra/data.
# Run [{color:#0066cc}nodetool
refresh{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRefresh.html].
h2. Restoring from centralized backups
This method uses
[{color:#0066cc}sstableloader{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/toolsBulkloader.html]
to restore snapshots.
# Verify that the SSTable version is compatible with the current version of
DSE:
## Locate the version in the file names.
Use the version number in the SSTable file name to determine compatibility and
upgrade requirements. The first two letters of the file name is the version,
where the first letter indicates a major version and the second letter
indicates a minor version. For example, the following SSTable version is
mc:{{data/cycling/cyclist_expenses-2d955621194c11e7a38d9504a063a84e/mc-6-big-Data.db}}
## Using the correct DSE version of {{sstableupgrade}}, to create a compatible
version:
|SSTable compatibility and upgrade version|
||DSE version||SSTable
version||[{color:#0066cc}sstableloader{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/toolsBulkloader.html]
supported
version||[{color:#0066cc}sstableupgrade{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/ToolsSSTableupgrade.html]
and [{color:#0066cc}nodetool
upgradesstables{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsUpgradeSstables.html]
supported version||
|4.0|jb|jb only (must upgrade to load)|i* only|
|4.5|
|4.6|
|4.7|ka|ka and jb|j* only|
|4.8|
|5.0.x|ma|ka and ma|k* only|
|5.0.x|mb|ka, ma, and mb|ka and ma|
|5.0.x|mc|ka, ma, mb, and mc|ka, ma, and mb|
|5.1.x|
# Make sure the table schema exists and is the same as when the snapshot was
created.The nodetool snapshot command creates a table schema in the output
directory. If the table does not exist, recreate it using the schema.cql file.
# If necessary,
[{color:#0066cc}truncate{color}|https://docs.datastax.com/en/dse/5.1/cql/cql/cql_reference/cql_commands/cqlTruncate.html]
the table.
Note: You may not need to truncate under certain conditions. For example, if a
node lost a disk, you might restart before restoring so that the node continues
to receive new writes before starting the restore procedure.
Truncating is usually necessary. For example, if there was an accidental
deletion of data, the tombstone from that delete has a later write timestamp
than the data in the snapshot. If you restore without truncating (removing the
tombstone), the database continues to shadow the restored data. This behavior
also occurs for other types of overwrites and causes the same problem.
# Restore the most recent snapshot using the
[{color:#0066cc}sstableloader{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/toolsBulkloader.html]
tool on the backed-up SSTables.
The sstableloader streams the SSTables to the correct nodes. You do not need
to remove the commitlogs or drain or restart the nodes.
h1. Restoring a snapshot into a new cluster
Suppose you want to copy a snapshot of SSTable data files from a three node
DataStax Enterprise cluster with vnodes enabled (128 tokens) and recover it on
another newly created three node cluster (128 tokens). The token ranges will
not match, because the token ranges cannot be exactly the same in the new
cluster. You need to specify the tokens for the new cluster that were used in
the old cluster.
Note: This procedure assumes you are familiar with [restoring a
snapshot|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupSnapshotRestore.html]
and configuring and [initializing a
cluster|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/production/initDSETOC.html].
h2. Procedure
To recover the snapshot on the new cluster:
# From the old cluster, retrieve the list of tokens associated with each
node's IP:
{color:#990073}$ {color}nodetool ring | grep ip_address_of_node | awk
{color:#4169e1}'\{print $NF ","}'{color} | xargs
# In the
[cassandra.yaml|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml]
file for each node in the new cluster, add the list of tokens you obtained in
the previous step to the
[initial_token|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configCassandra_yaml.html#configCassandra_yaml__initial_token]
parameter using the same {color:#374c51}num_tokens{color} setting as in the
old cluster.
# Make any other necessary changes in the new cluster's
[cassandra.yaml|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml]
and property files so that the new nodes match the old cluster settings. Make
sure the seed nodes are set for the new cluster.
# Clear the system table data from each new node:
{color:#990073}$ {color}{color:#0086b3}sudo{color} rm -rf
/var/lib/cassandra/data/system/*
This allows the new nodes to use the initial tokens defined in the
[cassandra.yaml|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml]
when they restart.
# Start each node using the specified list of token ranges in new cluster's
[cassandra.yaml|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsSnapshotRestoreNewCluster.html#opsSnapshotRestoreNewCluster__cassandrayaml]:
{{initial_token: -9211270970129494930, -9138351317258731895,
-8980763462514965928, ...}}
# Create schema in the new cluster. All the schemas from the old cluster must
be reproduced in the new cluster.
# [Stop the
node|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/startStop/startStopDseToc.html].
Using {{nodetool refresh}} is unsafe because files within the data directory
of a running node can be silently overwritten by identically named just-flushed
SSTables from memtable flushes or compaction. Copying files into the data
directory and restarting the node will not work for the same reason.
# [Restore the SSTable files
snapshotted|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupSnapshotRestore.html]
from the old cluster onto the new cluster using the same directories,
w{color:#374c51}hile noting that the UUID component of target directory names
has changed. Without restoration, the new cluster will not have data to read
upon restart. {color}
# {color:#374c51}[Restart the
node|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/startStop/startStopDseToc.html].{color}
Recovering from a single disk failure using JBOD
Steps for recovering from a single disk failure in a disk array using JBOD
(just a bunch of disks).
Steps for recovering from a single disk failure in a disk array using JBOD
(just a bunch of disks).
DataStax Enterprise might not fail from the loss of one disk in a JBOD array,
but some reads and writes may fail when:The operation's consistency level is
ALL.
The data being requested or written is stored on the defective disk.
The data to be compacted is on the defective disk.
It's possible that you can simply replace the disk, restart DataStax
Enterprise, and run nodetool repair. However, if the disk crash corrupted
system table, you must remove the incomplete data from the other disks in the
array. The procedure for doing this depends on whether the cluster uses vnodes
or single-token architecture.
Procedure
If a disk fails on a node in a cluster using DataStax Enterprise 5.0 or
earlier, replace the node.
Verify that the node has a defective disk and identify the disk, by checking
the logs on the affected node.
Disk failures are logged in FILE NOT FOUND entries, which identifies the mount
point or disk that has failed.
If the node is still running, stop DSE and shut down the node.
Replace the defective disk and restart the node.
If the node cannot restart:
Try restarting DataStax Enterprise without bootstrapping the node:
Package and Installer-Services installations: Add the following option to
cassandra-env.sh [file:JVM_OPTS=]"$JVM_OPTS
-Dcassandra.allow_unsafe_replace=true
Starting DataStax Enterprise as a service.
After the node bootstraps, remove the -Dcassandra.allow_unsafe_replace=true
parameter from cassandra-env.sh.
Starting DataStax Enterprise as a service.
Tarball and Installer-No Services installations:
Start DataStax Enterprise with this option:$ sudo bin/dse cassandra
Dcassandra.allow_unsafe_replace=true
Tarball and Installer No-Services path: installation_location
If DataStax Enterprise restarts, run nodetool repair on the node. If not,
replace the node.
If the repair succeeds, the node is restored to production. Otherwise, go to 7
or 8.
For a cluster using vnodes: On the affected node, clear the system directory
on each functioning drive.
Example for a node with a three disk JBOD array:$ -/mnt1/cassandra/data
$ -/mnt2/cassandra/data
$ -/mnt3/cassandra/data
If mnt1 has failed:$ rm -fr /mnt2/cassandra/data/system
$ rm -fr /mnt3/cassandra/data/system
Restart DataStax Enterprise without bootstrapping as described in 4:
$ -Dcassandra.allow_unsafe_replace=true
Run nodetool repair on the node.
If the repair succeeds, the node is restored to production. If not, replace the
dead node.
For a cluster single-token nodes: On one of the cluster's working nodes, run
nodetool ring to retrieve the list of the repaired node's tokens:
$ nodetool ring | grep ip_address_of_node | awk '
{print $NF ","}
' | xargs
Copy the output of the nodetool ring into a spreadsheet (space-delimited).
Edit the output, keeping the list of tokens and deleting the other columns.
On the node with the new disk, open the cassandra.yaml file and add the tokens
(as a comma-separated list) to the initial_token property.
Change any other non-default settings in the new nodes to match the existing
nodes. Use the diff command to find and merge any differences between the nodes.
If the repair succeeds, the node is restored to production. If not, replace the
node.
On the affected node, clear the system directory on each functioning drive.
Example for a node with a three disk JBOD array:$ -/mnt1/cassandra/data
$ -/mnt2/cassandra/data
$ -/mnt3/cassandra/data
If mnt1 has failed:$ rm -fr /mnt2/cassandra/data/system
$ rm -fr /mnt3/cassandra/data/system
Restart DataStax Enterprise without bootstrapping as described in 4:
$ -Dcassandra.allow_unsafe_replace=true
Run nodetool repair on the node.
If the repair succeeds, the node is restored to production. If not, replace the
node.
h1. Recovering from a single disk failure using JBOD
Steps for recovering from a single disk failure in a disk array using JBOD
(just a bunch of disks).
Steps for recovering from a single disk failure in a disk array using JBOD
(just a bunch of disks).
DataStax Enterprise might not fail from the loss of one disk in a JBOD array,
but some reads and writes may fail when: * The operation's consistency level is
ALL.
* The data being requested or written is stored on the defective disk.
* The data to be compacted is on the defective disk.
It's possible that you can simply replace the disk, restart DataStax
Enterprise, and run nodetool repair. However, if the disk crash corrupted
system table, you must remove the incomplete data from the other disks in the
array. The procedure for doing this depends on whether the cluster uses
[{color:#0066cc}vnodes{color}|https://docs.datastax.com/en/glossary/doc/glossary/gloss_vnode.html]
or single-token architecture.
h2. Procedure
If a disk fails on a node in a cluster using DataStax Enterprise 5.0 or
earlier, [{color:#0066cc}replace the
node{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsReplaceNode.html].
# Verify that the node has a defective disk and identify the disk, by checking
the
[{color:#0066cc}logs{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configLoggingLevels.html]
on the affected node.
Disk failures are logged in {{FILE NOT FOUND}} entries, which identifies the
mount point or disk that has failed.
# If the node is still running, [{color:#0066cc}stop
DSE{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/startStop/startStopDseToc.html]
and shut down the node.
# Replace the defective disk and restart the node.
# If the node cannot restart:
## Try restarting DataStax Enterprise without bootstrapping the node:
Package and Installer-Services installations: # Add the following option to
[{color:#0066cc}cassandra-env.sh{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__cassandraenvsh]
file:{{JVM_OPTS="$JVM_OPTS -Dcassandra.allow_unsafe_replace=true}}
# [{color:#0066cc}Starting DataStax Enterprise as a
service{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/startStop/startDseService.html].
# After the node bootstraps, remove the
{{-Dcassandra.allow_unsafe_replace=true}} parameter from cassandra-env.sh.
# [{color:#0066cc}Starting DataStax Enterprise as a
service{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/startStop/startDseService.html].
Tarball and Installer-No Services installations:
*** Start DataStax Enterprise with this option:
$ sudo bin/dse cassandra Dcassandra.allow_unsafe_replace=trueTarball and
Installer No-Services path: {{installation_location}}
# If DataStax Enterprise restarts, run [{color:#0066cc}nodetool
repair{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRepair.html]
on the node. If not, [{color:#0066cc}replace the
node{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsReplaceNode.html].
# If the repair succeeds, the node is restored to production. Otherwise, go to
[{color:#0066cc}7{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__vnode-used]
or
[{color:#0066cc}8{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__single-token-nodes-used].
# For a cluster using vnodes:
## On the affected node, clear the system directory on each functioning drive.
Example for a node with a three disk JBOD array:$ -/mnt1/cassandra/data
$ -/mnt2/cassandra/data
$ -/mnt3/cassandra/data
If {{mnt1}} has failed:
$ rm -fr /mnt2/cassandra/data/system
$ rm -fr /mnt3/cassandra/data/system
## Restart DataStax Enterprise without bootstrapping as described in
[{color:#0066cc}4{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__restart-w-out-boots]:
$ -Dcassandra.allow_unsafe_replace=true
## Run [{color:#0066cc}nodetool
repair{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRepair.html]
on the node.
If the repair succeeds, the node is restored to production. If not,
[{color:#0066cc}replace the dead
node{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsReplaceNode.html].
# For a cluster single-token nodes:
## On one of the cluster's working nodes, run [{color:#0066cc}nodetool
ring{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRing.html]
to retrieve the list of the repaired node's tokens:
$ nodetool ring | grep ip_address_of_node | awk ' \{print $NF ","}' | xargs
## Copy the output of the nodetool ring into a spreadsheet (space-delimited).
## Edit the output, keeping the list of tokens and deleting the other columns.
## On the node with the new disk, open the
[{color:#0066cc}cassandra.yaml{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__cassandrayaml]
file and add the tokens (as a comma-separated list) to the
[{color:#0066cc}initial_token{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configCassandra_yaml.html#configCassandra_yaml__initial_token]
property.
## Change any other non-default settings in the new nodes to match the
existing nodes. Use the diff command to find and merge any differences between
the nodes.
If the repair succeeds, the node is restored to production. If not,
[{color:#0066cc}replace the
node{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsReplaceNode.html].
## On the affected node, clear the system directory on each functioning drive.
Example for a node with a three disk JBOD array:$ -/mnt1/cassandra/data
$ -/mnt2/cassandra/data
$ -/mnt3/cassandra/data
If {{mnt1}} has failed:
$ rm -fr /mnt2/cassandra/data/system
$ rm -fr /mnt3/cassandra/data/system
## Restart DataStax Enterprise without bootstrapping as described in
[{color:#0066cc}4{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsRecoverUsingJBOD.html#opsRecoverUsingJBOD__restart-w-out-boots]:
$ -Dcassandra.allow_unsafe_replace=true
## Run [{color:#0066cc}nodetool
repair{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRepair.html]
on the node.
If the repair succeeds, the node is restored to production. If not,
[{color:#0066cc}replace the
node{color}|https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsReplaceNode.html].
cassandra.yaml
The location of the cassandra.yaml file depends on the type of installation:
|Package installations
Installer-Services installations|/etc/dse/cassandra/cassandra.yaml|
|Tarball installations
Installer-No Services
installations|installation_location/resources/cassandra/conf/cassandra.yaml|
cassandra-rackdc.properties
The location of the cassandra-rackdc.properties file depends on the type of
installation:
|Package installations
Installer-Services installations|/etc/dse/cassandra/cassandra-rackdc.properties|
|Tarball installations
Installer-No Services
installations|installation_location/resources/cassandra/conf/cassandra-rackdc.properties|
cassandra-topology.properties
The location of the cassandra-topology.properties file depends on the type of
installation:
|Package installations
Installer-Services
installations|/etc/dse/cassandra/cassandra-topology.properties|
|Tarball installations
Installer-No Services
installations|installation_location/resources/cassandra/conf/cassandra-topology.properties|
cassandra-env.sh
The location of the cassandra-env.sh file depends on the type of installation:
|Package installations
Installer-Services installations|/etc/dse/cassandra/cassandra-env.sh|
|Tarball installations
Installer-No Services
installations|installation_location/resources/cassandra/conf/cassandra-env.sh|
*
> The Backups web page on the web site is empty
> ---------------------------------------------
>
> Key: CASSANDRA-14272
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14272
> Project: Cassandra
> Issue Type: Improvement
> Components: Documentation and Website
> Reporter: Kenneth Brotman
> Priority: Major
>
> [http://cassandra.apache.org/doc/latest/operating/backups.html]
> is empty. Please contribute content. Myself or someone else will take it
> from there.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
