I come from the hadoop world where we have a cluster with probably over
500 drives. Drives fail all the time; or well several a year anyway.
We remove that single drive from HDFS, HDFS re-balances, and when we get
around to it, we swap in a new drive, format it, and add it back to
HDFS. We keep the OS drives separate from the data drives and ensure
that the OS volume is in a RAID mirror. It's painful when OS drives
fail, so mirror works. When space is low, we add another node with lots
of disks.
We are repurposing this same hardware to run a large Cassandra cluster.
I'd love it if Cassandra could support larger individual nodes, but
we've been trying to configure it with lots of disks for redundancy,
with the idea that we won't use an entire nodes storage only for
Cassandra. As was mentioned a long while back, blades seem to make more
sense for Cassandra than single nodes with lots of disk, but we've got
what we've got!
:)
So far, no issues with:
Stop node, remove drive from cassandra config, start node, run repair -
version 4.1.
-Joe
On 1/17/2023 10:11 AM, Durity, Sean R via user wrote:
For physical hardware when disks fail, I do a removenode, wait for the
drive to be replaced, reinstall Cassandra, and then bootstrap the node
back in (and run clean-up across the DC).
All of our disks are presented as one file system for data, which is
not what the original question was asking.
Sean R. Durity
*From:*Marc Hoppins <marc.hopp...@eset.com>
*Sent:* Tuesday, January 17, 2023 3:57 AM
*To:* user@cassandra.apache.org
*Subject:* [EXTERNAL] RE: Failed disks - correct procedure
HI all, I was pondering this very situation. We have a node with a
crapped-out disk (not the first time). Removenode vs repairnode: in
regard time, there is going to be little difference twixt replacing a
dead node and removing then re-installing
INTERNAL USE
HI all,
I was pondering this very situation.
We have a node with a crapped-out disk (not the first time).
Removenode vs repairnode: in regard time, there is going to be little
difference twixt replacing a dead node and removing then re-installing
a node. There is going to be a bunch of reads/writes and
verifications (or similar) which is going to take a similar amount of
time...or do I read that wrong?
For myself, I just go with removenode and then rejoin after HDD has
bee replaced. Usually the fix exceeds the wait time and the node is
then out of the system anyway.
-----Original Message-----
From: Joe Obernberger <joseph.obernber...@gmail.com>
Sent: Monday, January 16, 2023 6:31 PM
To: Jeff Jirsa <jji...@gmail.com>; user@cassandra.apache.org
Subject: Re: Failed disks - correct procedure
EXTERNAL
I'm using 4.1.0-1.
I've been doing a lot of truncates lately before the drive failed
(research project). Current drives have about 100GBytes of data each,
although the actual amount of data in Cassandra is much less (because
of truncates and snapshots). The cluster is not homo-genius; some
nodes have more drives than others.
nodetool status -r
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host
ID Rack
UN nyx.querymasters.com 7.9 GiB 250 ?
07bccfce-45f1-41a3-a5c4-ee748a7a9b98 rack1
UN enceladus.querymasters.com 6.34 GiB 200 ?
274a6e8d-de37-4e0b-b000-02d221d858a5 rack1
UN aion.querymasters.com 6.31 GiB 200 ?
59150c47-274a-46fb-9d5e-bed468d36797 rack1
UN calypso.querymasters.com 6.26 GiB 200 ?
e83aa851-69b4-478f-88f6-60e657ea6539 rack1
UN fortuna.querymasters.com 7.1 GiB 200 ?
49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
UN kratos.querymasters.com 6.36 GiB 200 ?
0d9509cc-2f23-4117-a883-469a1be54baf rack1
UN charon.querymasters.com 6.35 GiB 200 ?
d9702f96-256e-45ae-8e12-69a42712be50 rack1
UN eros.querymasters.com 6.4 GiB 200 ?
93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
UN ursula.querymasters.com 6.24 GiB 200 ?
4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
UN gaia.querymasters.com 6.28 GiB 200 ?
b2e5366e-8386-40ec-a641-27944a5a7cfa rack1
UN chaos.querymasters.com 3.78 GiB 120 ?
08a19658-40be-4e55-8709-812b3d4ac750 rack1
UN pallas.querymasters.com 6.24 GiB 200 ?
b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
UN paradigm7.querymasters.com 16.25 GiB 500 ?
1ccd2cc5-3ee5-43c5-a8c3-7065bdc24297 rack1
UN aether.querymasters.com 6.36 GiB 200 ?
352fd049-32f8-4be8-9275-68b145ac2832 rack1
UN athena.querymasters.com 15.85 GiB 500 ?
b088a8e6-42f3-4331-a583-47ef5149598f rack1
-Joe
On 1/16/2023 12:23 PM, Jeff Jirsa wrote:
> Prior to cassandra-6696 you’d have to treat one missing disk as a
> failed machine, wipe all the data and re-stream it, as a tombstone for
> a given value may be on one disk and data on another (effectively
> redirecting data)
>
> So the answer has to be version dependent, too - which version were you using?
>
>> On Jan 16, 2023, at 9:08 AM, Tolbert, Andy <x...@andrewtolbert.com> wrote:
>>
>> Hi Joe,
>>
>> Reading it back I realized I misunderstood that part of your email,
>> so you must be using data_file_directories with 16 drives? That's a
>> lot of drives! I imagine this may happen from time to time given
>> that disks like to fail.
>>
>> That's a bit of an interesting scenario that I would have to think
>> about. If you brought the node up without the bad drive, repairs are
>> probably going to do a ton of repair overstreaming if you aren't
>> using
>> 4.0 (https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-3200__;!!M-nmYVHPHQ!NmM-JuBX-GTYHt0XeaEWNz7saGfIvnRUEAy3HG6hX_i0bdaIzpo4ceBTx-mB1K9PsPJhfCb0ZCrgVxL7EkOS5AaQVTU$
<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/CASSANDRA-3200__;!!M-nmYVHPHQ!NmM-JuBX-GTYHt0XeaEWNz7saGfIvnRUEAy3HG6hX_i0bdaIzpo4ceBTx-mB1K9PsPJhfCb0ZCrgVxL7EkOS5AaQVTU$>)
which may
>> put things into a really bad state (lots of streaming = lots of
>> compactions = slower reads) and you may be seeing some inconsistency
>> if repairs weren't regularly running beforehand.
>>
>> How much data was on the drive that failed? How much data do you
>> usually have per node?
>>
>> Thanks,
>> Andy
>>
>>> On Mon, Jan 16, 2023 at 10:59 AM Joe Obernberger
>>> <joseph.obernber...@gmail.com> wrote:
>>>
>>> Thank you Andy.
>>> Is there a way to just remove the drive from the cluster and replace
>>> it later? Ordering replacement drives isn't a fast process...
>>> What I've done so far is:
>>> Stop node
>>> Remove drive reference from /etc/cassandra/conf/cassandra.yaml
>>> Restart node
>>> Run repair
>>>
>>> Will that work? Right now, it's showing all nodes as up.
>>>
>>> -Joe
>>>
>>>> On 1/16/2023 11:55 AM, Tolbert, Andy wrote:
>>>> Hi Joe,
>>>>
>>>> I'd recommend just doing a replacement, bringing up a new node with
>>>> -Dcassandra.replace_address_first_boot=ip.you.are.replacing as
>>>> described here:
>>>> https://urldefense.com/v3/__https://cassandra.apache.org/doc/4.1/cassandra/operating/topo_chang__;!!M-nmYVHPHQ!NmM-JuBX-GTYHt0XeaEWNz7saGfIvnRUEAy3HG6hX_i0bdaIzpo4ceBTx-mB1K9PsPJhfCb0ZCrgVxL7EkOSUkY8zuQ$
<https://urldefense.com/v3/__https:/cassandra.apache.org/doc/4.1/cassandra/operating/topo_chang__;!!M-nmYVHPHQ!NmM-JuBX-GTYHt0XeaEWNz7saGfIvnRUEAy3HG6hX_i0bdaIzpo4ceBTx-mB1K9PsPJhfCb0ZCrgVxL7EkOSUkY8zuQ$>
>>>> es.html#replacing-a-dead-node
>>>>
>>>> Before you do that, you will want to make sure a cycle of repairs
>>>> has run on the replicas of the down node to ensure they are
>>>> consistent with each other.
>>>>
>>>> Make sure you also have 'auto_bootstrap: true' in the yaml of the
>>>> node you are replacing and that the initial_token matches the node
>>>> you are replacing (If you are not using vnodes) so the node doesn't
>>>> skip bootstrapping. This is the default, but felt worth mentioning.
>>>>
>>>> You can also remove the dead node, which should stream data to
>>>> replicas that will pick up new ranges, but you also will want to do
>>>> repairs ahead of time too. To be honest it's not something I've
>>>> done recently, so I'm not as confident on executing that procedure.
>>>>
>>>> Thanks,
>>>> Andy
>>>>
>>>>
>>>> On Mon, Jan 16, 2023 at 9:28 AM Joe Obernberger
>>>> <joseph.obernber...@gmail.com> wrote:
>>>>> Hi all - what is the correct procedure when handling a failed disk?
>>>>> Have a node in a 15 node cluster. This node has 16 drives and
>>>>> cassandra data is split across them. One drive is failing. Can I
>>>>> just remove it from the list and cassandra will then replicate? If not - what?
>>>>> Thank you!
>>>>>
>>>>> -Joe
>>>>>
>>>>>
>>>>> --
>>>>> This email has been checked for viruses by AVG antivirus software.
>>>>> https://urldefense.com/v3/__http://www.avg.com__;!!M-nmYVHPHQ!NmM-JuBX-GTYHt0XeaEWNz7saGfIvnRUEAy3HG6hX_i0bdaIzpo4ceBTx-mB1K9PsPJhfCb0ZCrgVxL7EkOSyg3IaPU$
<https://urldefense.com/v3/__http:/www.avg.com__;!!M-nmYVHPHQ!NmM-JuBX-GTYHt0XeaEWNz7saGfIvnRUEAy3HG6hX_i0bdaIzpo4ceBTx-mB1K9PsPJhfCb0ZCrgVxL7EkOSyg3IaPU$>
>>> --
>>> This email has been checked for viruses by AVG antivirus software.
>>> https://urldefense.com/v3/__http://www.avg.com__;!!M-nmYVHPHQ!NmM-JuBX-GTYHt0XeaEWNz7saGfIvnRUEAy3HG6hX_i0bdaIzpo4ceBTx-mB1K9PsPJhfCb0ZCrgVxL7EkOSyg3IaPU$
<https://urldefense.com/v3/__http:/www.avg.com__;!!M-nmYVHPHQ!NmM-JuBX-GTYHt0XeaEWNz7saGfIvnRUEAy3HG6hX_i0bdaIzpo4ceBTx-mB1K9PsPJhfCb0ZCrgVxL7EkOSyg3IaPU$>
--
This email has been checked for viruses by AVG antivirus software.
https://urldefense.com/v3/__http://www.avg.com__;!!M-nmYVHPHQ!NmM-JuBX-GTYHt0XeaEWNz7saGfIvnRUEAy3HG6hX_i0bdaIzpo4ceBTx-mB1K9PsPJhfCb0ZCrgVxL7EkOSyg3IaPU$
<https://urldefense.com/v3/__http:/www.avg.com__;!!M-nmYVHPHQ!NmM-JuBX-GTYHt0XeaEWNz7saGfIvnRUEAy3HG6hX_i0bdaIzpo4ceBTx-mB1K9PsPJhfCb0ZCrgVxL7EkOSyg3IaPU$>
--
This email has been checked for viruses by AVG antivirus software.
www.avg.com
