Re: [DRBD-user] Eraser coding
On 2021-09-24 03:16, Roland Kammerer wrote: On Wed, Sep 22, 2021 at 12:19:57PM +, Carmelo McCutcheon wrote: How do you get a version that supports eraser coding? "eraser" should be easy, use shred(1) :D There has been some work in the direction of erasure coding but that would be something for the far future (please don't ask when it will be ready). I need to run a 3 node cluster active/active/active…. Good luck (but not with DRBD alone). Regards, rck Could you use DRBD in active/passive mode on two nodes backing NFS, and then mount the NFS mount on the three nodes? Avoids the need for a cluster FS (and it's expensive cluster locking), and avoids the unsupported active/active(/active). -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] protocol C replication - unexpected behaviour
On 2021-08-10 3:16 p.m., Janusz
Jaskiewicz wrote:
Hi,
Thanks for your answers.
Answering your questions:
DRBD_KERNEL_VERSION=9.0.25
Linux kernel:
4.18.0-305.3.1.el8.x86_6
File system type:
XFS.
So the file system is not cluster-aware, but as far as I
understand in an active/passive setup - single primary (that I
have) it should be OK.
Just checked the doc which seems to confirm that.
I think the problem may come from the way I'm testing it.
I came up with this testing scenario, that I described in
my first post, because I didn't have an easy way to abruptly
restart the server.
When I do the hard reset of the primary server it works as
expected (at least I can find a logical explanation).
I think what happened in my previous scenario was:
Service is writing to the disk, and some portion of the
written data is in a disk cache. As the picture https://linbit.com/wp-content/uploads/drbd/drbd-guide-9_0-en/images/drbd-in-kernel.png
shows, the cache is above the DRBD module.
Then I kill the service and the network, but some data is
still in the cache.
At some point the cache is flushed and the data gets
written to the disk.
DRBD probably reports some error at this point, as it can't
send that data to the secondary node (DRBD thinks the other
node has left the cluster).
When I check the files at this point I see more data on the
primary because it also contains the data from the cache,
which were not replicated because the network was down when
the data hit the DRBD.
When I do the hard restart of the server, data in the cache
is lost, so we don't observe the result as above.
Does it make sense?
Regards,
Janusz.
OK, it sounded from your first post like you have the FS mounted
on both nodes at the same time, that would be a problem. If it's
only mounted in one place at a time, then it's OK.
As for caching; DRBD on the Secondary will say "write complete"
to the primary, in protocol C, when it has been told that the disk
write is complete. So if the cache is _above_ drbd's kernel
module, then that's probably not the problem because the Secondary
won't tell the primary it's done until it receives the data. If
there is a caching issue _below_ DRBD on the Secondary, then it's
_possible_ that's the problem, but I doubt it. The reason is that
whatever is managing the cache below DRBD on the Secondary should
know that a given block hasn't flushed yet and, on read request,
read from cache not disk. This is a guess on my part.
What are your 'disk { disk-flushes [yes|no]; and md-flushes
[yes|no]; }' set to?
--
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould
___
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] protocol C replication - unexpected behaviour
On 2021-08-10 2:11 a.m., Eddie Chapman wrote: On 10/08/2021 00:01, Digimer wrote: On 2021-08-05 5:53 p.m., Janusz Jaskiewicz wrote: Hello. I'm experimenting a bit with DRBD in a cluster managed by Pacemaker. It's a two node, active-passive cluster and the service that I'm trying to put in the cluster writes to the file system. The service manages many files, it appends to some of them and increments the single integer value in others. I'm observing surprising behaviour and I would like to ask you if what I see is expected or not (I think not). I'm using protocol C, but still I see some delay in the files that are being replicated to the secondary server. For the files that increment the integer I see a difference which corresponds roughly to 1 second of traffic. I'm really surprised to see this, as protocol C should guarantee synchronous replication. I'd rather expect some delay in processing (potentially slower disk writes due to the network replication). The way I'm testing it: The service runs on primary and writes to DRBD drive, secondary connected and "UpToDate". I kill the service abruptly (kill -9) and then take down the network interface between primary and secondary (kill and ifdown commands in the script so executed quite promptly one after the other). Then I mount the DRBD drive on both nodes and check the difference in the files with incrementing integer. I would appreciate any help or pointers on how to fix this. But first of all I would like to confirm that this behaviour is not expected. Also if it is expected/allowed, how can I decrease the impact? What filesystem are you using? Is it cluster / multi-node aware? The filesystem may be relevant in that filesystems can behave in ways one might not expect, depending on how they are tuned, so would be good to know what the fs is and what mount options are being used. However, the filesystem certainly does not need to be aware that the underlying block device is drbd or a cluster of any kind. A drbd device should look like a regular block device and there is no need to treat it like anything else. I would like to know the kernel and drbd versions, if these are old enough then expecting things to "work" in a sane fashion might not be a reasonable expectation :-) The reason I asked is because all DRBD does is replicate the blocks. It doesn't (and can't) handle locks to avoid corruption, update file lists, etc. If you've mounted, say, xfs or ext4 on both nodes, then it's a surprise it updates at all, never mind being a few seconds delayed. If you're using a proper cluster FS like GFS2 or ocfs, then we may want to investigate those. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] protocol C replication - unexpected behaviour
On 2021-08-05 5:53 p.m., Janusz
Jaskiewicz wrote:
Hello.
I'm experimenting a bit with DRBD in a cluster managed by
Pacemaker.
It's a two node, active-passive cluster and the service that I'm
trying to put in the cluster writes to the file system.
The service manages many files, it appends to some of them and
increments the single integer value in others.
I'm observing surprising behaviour and I would like to ask you
if what I see is expected or not (I think not).
I'm using protocol C, but still I see some delay in the files
that are being replicated to the secondary server.
For the files that increment the integer I see a difference
which corresponds roughly to 1 second of traffic.
I'm really surprised to see this, as protocol C should guarantee
synchronous replication.
I'd rather expect some delay in processing (potentially slower
disk writes due to the network replication).
The way I'm testing it:
The service runs on primary and writes to DRBD drive, secondary
connected and "UpToDate".
I kill the service abruptly (kill -9) and then take down the
network interface between primary and secondary (kill and ifdown
commands in the script so executed quite promptly one after the
other).
Then I mount the DRBD drive on both nodes and check the
difference in the files with incrementing integer.
I would appreciate any help or pointers on how to fix this.
But first of all I would like to confirm that this behaviour is
not expected.
Also if it is expected/allowed, how can I decrease the impact?
Please find below my configuration:
global_common.conf:
global {
usage-count yes;
udev-always-use-vnr; # treat implicit the same as explicit
volumes
}
common {
handlers {}
startup {}
options {}
disk {}
net {
protocol C;
}
}
my_resource.conf:
resource wyr-dim-persistence {
on dimprod01.my.clustertest.com {
device /dev/drbd1;
disk /dev/mapper/drbdpool-drbdata;
address 192.168.0.1:7789;
meta-disk internal;
}
on dimprod02.my.clustertest.com {
device /dev/drbd1;
disk /dev/mapper/drbdpool-drbdata;
address 192.168.0.2:7789;
meta-disk internal;
}
}
What filesystem are you using? Is it cluster / multi-node aware?
--
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould
___
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] The Problem of File System Corruption w/DRBD
On 2021-06-03 3:35 p.m., Eric Robinson wrote: >> Even this approach just moves the SPOF up from the FS to the SQL engine. >> >> The problem here is that you're still confusing redundancy with data >> integrity. To avoid data corruption, you need a layer that understands your >> data at a sufficient level to know what corruption looks like. Data >> integrity is >> yet another topic, and still separate from HA. >> > >> DRBD, and other HA tools, don't analyze the data, and nor should they >> (imagine the security and privacy concerns that would open up). If the HA >> layer is given data to replicate, it's job is to faithfully and accurately >> replicate >> the data. >> > > It seems like the two are sometimes intertwined. If GFS2, for example, about > integrity or redundancy? But I'm not really asking how to prevent filesystem > corruption. I'm asking (perhaps stupidly) the best/easiest way to make a > filesystem redundant. GFS2 coordinates access between nodes, to ensure no two step on each others blocks and that all know when to update their view of the FS. It is still above the redundancy layer, it is still just a file system at the end of the day. If, for example, you were writing data to an FS on top of DRBD, and one of the node's local storage started failing, the kernel would (should) inform the DRBD driver that there has been an IO error. In such a case, the DRBD device should detach from the local store and go diskless. All further read/writes on that node would (transparently) go to/from another node. In this way, I think, you get as close to the goal you're describing. In such a case though, you survived a hardware failure, _exactly_ what HA is all about. You would have no data loss and your managers would be happy. However, note how this example was below the data structure... It involved the detection of a hardware fault and mitigation of that fault. DRBD (like a RAID array) has no concept of data structures. So if something at the logic layer wrote bad data (ie: a user's deletion or saving of bad data), DRBD (again, like a RAID array) only cares to ensure that the data is on both/all nodes, byte for byte accurate. This is where the role of HA ends, and the role of anti-virus, security and data integrity / backups kick in. >> I think the real solution is not technical, it's expectations management. >> Your >> managers need to understand what each part of their infrastructure does >> and does not do. This way, if the concerns around data corruption are >> sufficient, they can invest in tools to protect the data integrity at the >> logical >> layer. >> >> HA protects against component failure. That's it's job, and it does it well, >> when well implemented. >> > > The filesystem is not a hardware component, but it is a cluster resource. The > other cluster resources are redundant, with that sole exception. I'm just > looking for a way around that problem. If there isn't one, then there isn't. Consider the example of a virtual machine running on top of DRBD / pacemaker (a setup I am very familiar with). If the host hardware fails, the VM can be preventatively migrated or recovered on the peer node. In this way, the data was preserved (up to the point of failure / reboot), and services are restored promptly. This was possible because, byte for byte the data was written to both host nodes. Voila! Full protection against hardware faults. Consider now that your VM gets hit with a cryptolocker virus. That attack is, faithfully, replicated to both nodes (exactly as it would replicate to both hard drives in a RAID 1 array). In this case, you're out of luck. Why? Because HA doesn't protect data integrity, it can't. It's role is to protect against hardware faults. This is true of the filesystem inside a VM, or a file system directly on top of a DRBD resource. The key take-away here is the role of different technologies in your over-all corporate resilience planning. It's one (very powerful) tool in a toolbox to protect your services and data. Backups, DR and anti-malware all play each their own roles in the big-picture planning. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] The Problem of File System Corruption w/DRBD
On 2021-06-03 11:09 a.m., Robert Sander wrote: > Hi, > > Am 03.06.21 um 14:50 schrieb Eric Robinson: > >> Yes, thanks, I've said for many years that HA is not a replacement for >> disaster recovery. Still, it is better to avoid downtime than to recover >> from it, and one of the main ways to achieve that is through redundancy, >> preferably a shared-nothing approach. If I have a cool 5-node cluster and >> the whole thing goes down because the filesystem gets corrupted, I can >> restore from backup, but management is going to wonder why a 5-node cluster >> could not provide availability. So the question remains: how to eliminate >> the filesystem as the SPOF? > > Then eliminate the shared filesystem and replicate data on application > level. > > - MySQL has Galera > - Dovecot has dsync > > Regards Even this approach just moves the SPOF up from the FS to the SQL engine. The problem here is that you're still confusing redundancy with data integrity. To avoid data corruption, you need a layer that understands your data at a sufficient level to know what corruption looks like. Data integrity is yet another topic, and still separate from HA. DRBD, and other HA tools, don't analyze the data, and nor should they (imagine the security and privacy concerns that would open up). If the HA layer is given data to replicate, it's job is to faithfully and accurately replicate the data. I think the real solution is not technical, it's expectations management. Your managers need to understand what each part of their infrastructure does and does not do. This way, if the concerns around data corruption are sufficient, they can invest in tools to protect the data integrity at the logical layer. HA protects against component failure. That's it's job, and it does it well, when well implemented. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] The Problem of File System Corruption w/DRBD
On 2021-06-02 5:17 p.m., Eric Robinson wrote: > Since DRBD lives below the filesystem, if the filesystem gets corrupted, > then DRBD faithfully replicates the corruption to the other node. Thus > the filesystem is the SPOF in an otherwise shared-nothing architecture. > What is the recommended way (if there is one) to avoid the filesystem > SPOF problem when clusters are based on DRBD? > > -Eric To start, HA, like RAID, is not a replacement for backups. That is the answer to a situation like this... HA (and other availability systems like RAID) protect against component failure. If a node fails, the peer recovers automatically and your services stay online. That's what DRBD and other HA solutions strive to provide; uptime. If you want to protect against corruption (accidental or intentional, a-la cryptolockers), you need a robust backup system to _compliment_ your HA solution. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] drbd-9.0.26
t the end of the chain could still have an outdated disk >(better than inconsistent) > * reduce lock contention on the secondary for many resources; can improve >performance significantly > * fix online verify to not clamp disk states to UpToDate > * fix promoting resync-target nodes; the problem was that it could modify >the bitmap of an ongoing resync; which leads to alarming log messages > * allow force primary on a sync-target node by breaking the resync > * fix adding of new volumes to resources with a primary node > * reliably detect split brain situation on both nodes > * improve error reporting for failures during attach > * implement 'blockdev --setro' in DRBD > * following upstream changes to DRBD up to Linux 5.10 and ensure >compatibility with Linux 5.8, 5.9, and 5.10 > > > https://www.linbit.com/downloads/drbd/9.0/drbd-9.0.26-1.tar.gz > https://github.com/LINBIT/drbd/commit/8e0c552326815d9d2bfd1cfd93b23f5692d7109c Thanks for this release! We've just updated and will report back if we have any issues. Cheers! digimer -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] : Backup "off line"
On 2020-10-12 5:36 a.m., Anthony Frnog wrote: > Hi, > > To begin, my apologies if I post à the wrong place. > > I use DRBD (version 9) on differents servers. My DRDB cluster is a > shared storage for VM on Proxmox et Vmware Currently, there is a lot of > ransomware attacks. This is a really problem... So, if my DRBD cluster > is encrypted, all data will be encrypted and all VMs will be "dead". My > question is: Is there a DRBD solution able to save all data stored on > DRDB node in order to restart my production if I have a ransowmare ? > > > Best regards Anthony There's no specific anti-ransomware tools in DRBD, but you could set something up easily enough. You could, for example, take periodic snapshots of the backing LVM devices (assuming you use LVs to back DRBD resources). You could keep N-number of snapshots and automatically cycle them out. How often you snapshot, and how many you keep, would depend on your wants and resources. You probably want to be able to roll back at least a week though, as it is my experience that some ransomware attacks lay dormant for a period of time before encrypting (to get into backups). In the end, DRBD is fundamentally an availability solution, and not a backup solution. (Same idea as how "RAID is not backup"). You really need to be sure that your data is backed up safely and incrementally. Any snapshot-based approach should be seen as a way to more rapidly recover to production, and not as a core backup method. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
[DRBD-user] Weird systemd / pacemaker stop crash issue
DRBD 9.0.23, utils 9.13, RHEL 8.
I've found a really odd issue where stopping pacemaker causes
'systemctl stop drbd' to hang. This is with drbd configured as a systemd
resource and with no DRBD resources yet configured (in DRBD or
pacemaker). The global-common.conf is standard (as created when installed).
In short;
While pacemaker is running, the DRBD resource can be stopped and
started fine. However, if the DRBD resource is running when you stop the
cluster, it hangs. Once this hang happens, even outside pacemaker, you
can't stop drbd ('systemctl stop drbd' hangs when called in another
terminal).
Now here is where it gets really weird...
If you stop the DRBD resource in pacemaker, then stop the cluster,
it's fine. More over, the crash never happens again. You can start the
cluster back up, then stop it with DRBD running and the daemon stops
cleanly.
To recreate the crash, I destroy the pacemaker cluster and reconfigure
systemd:drbd and the crash returns until one stop of the cluster with
DRBD already stopped, then the crash doesn't happen again.
Below are links to a series of pacemaker.log files (with debugging
on). The first pair starts with the initial config of pacemaker up to
the crash on shutdown. The second was fixing a stonith issue and then
repeating the crash. The third is a clean start to crash. The fourth
shows that drbd can be stopped and started with pacemaker, and still
crash. The last is where drbd is stopped when pacemaker stops, which
doesn't crash.
After this, everything works fine from then on.
Initial pacemaker config to DRBD crash;
node 1 - https://pastebin.com/raw/7QQGHW5g
node 2 - https://pastebin.com/raw/a1BYVzM7
Fix fence issue, repeat test, DRBD crash;
node 1 - https://pastebin.com/raw/1fZH1SSP
nod2 2 - https://pastebin.com/raw/p7i3absC
Fresh cluster start, crash on stop;
node 1 - https://pastebin.com/raw/gYNiMHDC
node 2 - https://pastebin.com/raw/VzkaPyEb
DRBD resource stopped, started, stop pacemaker, crash;
node 1 - https://pastebin.com/raw/LDTpmncY
node 2 - https://pastebin.com/raw/ryj2J6Qt
DRBD stopped resource, stop cluster, start cluster, stop cluster OK
node 1 - https://pastebin.com/raw/haECJz8y
node 2 - https://pastebin.com/raw/tBSD0ZyJ
--
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of
Einstein’s brain than in the near certainty that people of equal talent
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould
___
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] secondary not getting promoted to primary
The best advice would be to first upgrade (or create a new test system) with the latest 8.4 release, and see if the problem remains and can be reproduced. digimer On 2020-03-11 9:47 a.m., Mona Sinha wrote: > Hello Roland, > > Thanks for recognising me . But on a serious note, i have seen this > incident on newer versions of drbd too. I am very keen on learning new > things about drbd. Please help me as much as you can. > > Regards, > Mona > > > On Wed, 11 Mar, 2020, 6:55 PM Roland Kammerer, > mailto:roland.kamme...@linbit.com>> wrote: > > Hi, > > I remember you... Is this again some DRBD 8.3 that is EoL for a decade? > > Regards, rck > ___ > Star us on GITHUB: https://github.com/LINBIT > drbd-user mailing list > drbd-user@lists.linbit.com <mailto:drbd-user@lists.linbit.com> > https://lists.linbit.com/mailman/listinfo/drbd-user > > > ___ > Star us on GITHUB: https://github.com/LINBIT > drbd-user mailing list > drbd-user@lists.linbit.com > https://lists.linbit.com/mailman/listinfo/drbd-user > -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] How long does it take OOS to clear
Without knowing your setup or what is limiting you; I can suggest two
options;
1. Faster hardware (links speed / peer disk)
2. Switch to Protocol C
digimer
On 2019-10-21 4:36 p.m., G C wrote:
> Is there anything that will force the OOS to push what is out of sync?
>
>
> On Mon, Oct 21, 2019 at 11:00 AM Digimer <mailto:li...@alteeve.ca>> wrote:
>
> Tuning is quite instance-specific. I would always suggest starting by
> commenting out all tuning, see how it behaves, then tune. Premature
> optimization never is.
>
> digimer
>
> On 2019-10-21 1:31 p.m., G C wrote:
> > Would any of these values being changed help or would it need to
> be the
> > actual speed between the two nodes that needs to be increased?
> >
> > disk {
> > on-io-error detach;
> > c-plan-ahead 10;
> > c-fill-target 24M;
> > c-min-rate 80M;
> > c-max-rate 720M;
> > }
> > net {
> > protocol A;
> > max-buffers 36k;
> > sndbuf-size 1024k;
> > rcvbuf-size 2048k;
> > }
> >
> > Thank you
> >
> >
> >
> > On Mon, Oct 21, 2019 at 10:10 AM Digimer <mailto:li...@alteeve.ca>
> > <mailto:li...@alteeve.ca <mailto:li...@alteeve.ca>>> wrote:
> >
> > I assumed it wasn't paused, but that confirms it.
> >
> > Protocol A allows for out of sync to grow. It says "when the
> data in on
> > the network buffer to send to the peer, consider the write
> complete". As
> > such, data that hasn't made it over to the peer causes oos to
> climb. If
> > you have a steady write rate that is faster than your transmit
> > bandwidth, then seeing fairly steady OOS makes sense.
> >
> > To "fix" it, you need to increase the connection speed to the
> peer node.
> > Or, less likely, if the peer's disk is slower than the bandwidth
> > connecting it, speed up the disk write speed.
> >
> > In either case, what you are seeing is not a surprise, and
> it's not a
> > problem with DRBD. The only other option is to use protocol C,
> so that a
> > write isn't complete until it reaches the peer, but that will
> slow down
> > the write performance of the primary node to be whatever speed
> you have
> > to the peer. That's likely unacceptable.
> >
> > In short, you have a hardware/resource issue.
> >
> > digimer
> >
> > On 2019-10-21 12:19 p.m., G C wrote:
> > > version: 8.4.10
> > > Ran the resume-sync all and received:
> > > 0: Failure: (135) Sync-pause flag is already cleared
> > > Command 'drbdsetup-84 resume-sync 0' terminated with exit
> code 10
> > >
> > > Protocol used is 'A', our systems are running on a cloud
> environment.
> > >
> > >
> > >
> > >
> > > On Mon, Oct 21, 2019 at 9:09 AM Digimer <mailto:li...@alteeve.ca>
> > <mailto:li...@alteeve.ca <mailto:li...@alteeve.ca>>
> > > <mailto:li...@alteeve.ca <mailto:li...@alteeve.ca>
> <mailto:li...@alteeve.ca <mailto:li...@alteeve.ca>>>> wrote:
> > >
> > > 8.9.2 is the utils version, what is the kernel module
> version?
> > > (8.3.x/8.4.x/9.0.x)?
> > >
> > > It's possible something paused sync, but I doubt it. You
> can try
> > > 'drbdadm resume-sync all'. The oos number should change
> > constantly, any
> > > time a block changes it should go up and every time a block
> > syncs it
> > > should go down.
> > >
> > > What protocol are you using? A, B or C?
> > >
> > > digimer
> >
> >
> > --
> > Digimer
> > Papers and Projects: https://alteeve.com/w/
> > "I am, somehow, less interested in the weight and convolutions of
> > Einstein’s brain than in the near certainty that people of
> equal talent
> > have lived and died in cotton fields and sweatshops." -
>
Re: [DRBD-user] How long does it take OOS to clear
Tuning is quite instance-specific. I would always suggest starting by
commenting out all tuning, see how it behaves, then tune. Premature
optimization never is.
digimer
On 2019-10-21 1:31 p.m., G C wrote:
> Would any of these values being changed help or would it need to be the
> actual speed between the two nodes that needs to be increased?
>
> disk {
> on-io-error detach;
> c-plan-ahead 10;
> c-fill-target 24M;
> c-min-rate 80M;
> c-max-rate 720M;
> }
> net {
> protocol A;
> max-buffers 36k;
> sndbuf-size 1024k;
> rcvbuf-size 2048k;
> }
>
> Thank you
>
>
>
> On Mon, Oct 21, 2019 at 10:10 AM Digimer <mailto:li...@alteeve.ca>> wrote:
>
> I assumed it wasn't paused, but that confirms it.
>
> Protocol A allows for out of sync to grow. It says "when the data in on
> the network buffer to send to the peer, consider the write complete". As
> such, data that hasn't made it over to the peer causes oos to climb. If
> you have a steady write rate that is faster than your transmit
> bandwidth, then seeing fairly steady OOS makes sense.
>
> To "fix" it, you need to increase the connection speed to the peer node.
> Or, less likely, if the peer's disk is slower than the bandwidth
> connecting it, speed up the disk write speed.
>
> In either case, what you are seeing is not a surprise, and it's not a
> problem with DRBD. The only other option is to use protocol C, so that a
> write isn't complete until it reaches the peer, but that will slow down
> the write performance of the primary node to be whatever speed you have
> to the peer. That's likely unacceptable.
>
> In short, you have a hardware/resource issue.
>
> digimer
>
> On 2019-10-21 12:19 p.m., G C wrote:
> > version: 8.4.10
> > Ran the resume-sync all and received:
> > 0: Failure: (135) Sync-pause flag is already cleared
> > Command 'drbdsetup-84 resume-sync 0' terminated with exit code 10
> >
> > Protocol used is 'A', our systems are running on a cloud environment.
> >
> >
> >
> >
> > On Mon, Oct 21, 2019 at 9:09 AM Digimer <mailto:li...@alteeve.ca>
> > <mailto:li...@alteeve.ca <mailto:li...@alteeve.ca>>> wrote:
> >
> > 8.9.2 is the utils version, what is the kernel module version?
> > (8.3.x/8.4.x/9.0.x)?
> >
> > It's possible something paused sync, but I doubt it. You can try
> > 'drbdadm resume-sync all'. The oos number should change
> constantly, any
> > time a block changes it should go up and every time a block
> syncs it
> > should go down.
> >
> > What protocol are you using? A, B or C?
> >
> > digimer
>
>
> --
> Digimer
> Papers and Projects: https://alteeve.com/w/
> "I am, somehow, less interested in the weight and convolutions of
> Einstein’s brain than in the near certainty that people of equal talent
> have lived and died in cotton fields and sweatshops." - Stephen Jay
> Gould
>
--
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of
Einstein’s brain than in the near certainty that people of equal talent
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould
___
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] How long does it take OOS to clear
I assumed it wasn't paused, but that confirms it. Protocol A allows for out of sync to grow. It says "when the data in on the network buffer to send to the peer, consider the write complete". As such, data that hasn't made it over to the peer causes oos to climb. If you have a steady write rate that is faster than your transmit bandwidth, then seeing fairly steady OOS makes sense. To "fix" it, you need to increase the connection speed to the peer node. Or, less likely, if the peer's disk is slower than the bandwidth connecting it, speed up the disk write speed. In either case, what you are seeing is not a surprise, and it's not a problem with DRBD. The only other option is to use protocol C, so that a write isn't complete until it reaches the peer, but that will slow down the write performance of the primary node to be whatever speed you have to the peer. That's likely unacceptable. In short, you have a hardware/resource issue. digimer On 2019-10-21 12:19 p.m., G C wrote: > version: 8.4.10 > Ran the resume-sync all and received: > 0: Failure: (135) Sync-pause flag is already cleared > Command 'drbdsetup-84 resume-sync 0' terminated with exit code 10 > > Protocol used is 'A', our systems are running on a cloud environment. > > > > > On Mon, Oct 21, 2019 at 9:09 AM Digimer <mailto:li...@alteeve.ca>> wrote: > > 8.9.2 is the utils version, what is the kernel module version? > (8.3.x/8.4.x/9.0.x)? > > It's possible something paused sync, but I doubt it. You can try > 'drbdadm resume-sync all'. The oos number should change constantly, any > time a block changes it should go up and every time a block syncs it > should go down. > > What protocol are you using? A, B or C? > > digimer -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] How long does it take OOS to clear
8.9.2 is the utils version, what is the kernel module version? (8.3.x/8.4.x/9.0.x)? It's possible something paused sync, but I doubt it. You can try 'drbdadm resume-sync all'. The oos number should change constantly, any time a block changes it should go up and every time a block syncs it should go down. What protocol are you using? A, B or C? digimer On 2019-10-21 11:31 a.m., G C wrote: > I'm seeing OOS not being cleared for many days if not weeks, i.e. the > OOS number stays the same. > > Is there a way to tell if the blocks that are OOS are changing or if > it's the same ones that are just taking a very long time to sync with > the secondary? > > Version: 8.9.2 > > On Mon, Oct 7, 2019 at 7:55 AM Digimer <mailto:li...@alteeve.ca>> wrote: > > Out of Sync is a question of what has changed locally versus what has > reached the peer. It seems like you're generating changes on the Primary > faster than the Secondary can receive them. So one answer is to speed up > your replication link and/or the speed of the storage on the Secondary, > depending on which is the bottle-neck. > > The other option is to switch to "Protocol C", which tells DRBD to not > consider a write complete until it has reached both nodes. This will > effectively slow down your Primary node's storage to whatever speed the > Secondary can be written to, however, and may not be acceptable in your > use case (see back to point 1 above). > > digimer > > On 2019-10-07 10:40 a.m., G C wrote: > > I have an instance that seems to get OOS down lower and once in a > while > > it hits 0 but not very often. Typically my oos is about > 18-20, > > is there a way to clear this outside of the verify which takes a long > > time? Is there something deeper I can dig into that would stop > the oos > > from occurring? > > > > Thank you > > > > > > > > ___ > > Star us on GITHUB: https://github.com/LINBIT > > drbd-user mailing list > > drbd-user@lists.linbit.com <mailto:drbd-user@lists.linbit.com> > > https://lists.linbit.com/mailman/listinfo/drbd-user > > > > > -- > Digimer > Papers and Projects: https://alteeve.com/w/ > "I am, somehow, less interested in the weight and convolutions of > Einstein’s brain than in the near certainty that people of equal talent > have lived and died in cotton fields and sweatshops." - Stephen Jay > Gould > -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] drbd sycing again from the start
8.4.4 is old. Can you upgrade to the latest 8.4.11? I believe 8.4.4 was older than the other reporters with a similar issue, so this may be fixed. Upgrading to .11 should not cause any issues. PS - Please keep replies on the list. These discussions help others by being in the archives. digimer On 2019-10-17 9:54 a.m., Paras pradhan wrote: > drbd version is drbd-8.4.4-0.27.4.2 and yes are upgrading it to version > 9 in the near future. > > No it is not a live snapshot. Both drbd nodes were shutdown and used > clonezilla bootable image to take backup and also to restore. > > Thanks > Paras. > > On Wed, Oct 16, 2019 at 7:26 PM Digimer <mailto:li...@alteeve.ca>> wrote: > > I don't see the version, but looking in the mailing list archives, the > common recommendation is to upgrade. What version of DRBD 8 are you > using, exactly? > > Does the resync happen only after recovery? Is the backups of the nodes > done via live-snapshot? If so, then if there is _any_ time between the > two nodes being snapped, the UUID will differ and could be causing this. > > digimer > > On 2019-10-16 10:04 a.m., Paras pradhan wrote: > > Hi > > > > Here is the log for one of the drbd resource (which is 300GB). > > > > -- > > [ 194.780377] block drbd1: disk( Diskless -> Attaching ) > > [ 194.780536] block drbd1: max BIO size = 1048576 > > [ 194.780548] block drbd1: drbd_bm_resize called with capacity == > 629126328 > > [ 194.783069] block drbd1: resync bitmap: bits=78640791 words=1228763 > > pages=2400 > > [ 194.783077] block drbd1: size = 300 GB (314563164 KB) > > [ 194.793958] block drbd1: bitmap READ of 2400 pages took 3 jiffies > > [ 194.796342] block drbd1: recounting of set bits took additional > 1 jiffies > > [ 194.796348] block drbd1: 0 KB (0 bits) marked out-of-sync by on > disk > > bit-map. > > [ 194.796359] block drbd1: disk( Attaching -> Outdated ) > > [ 194.796366] block drbd1: attached to UUIDs > > 56E4CF14A115440C::02DCAB23D758DA48:02DBAB23D758DA49 > > [ 475.740272] block drbd1: drbd_sync_handshake: > > [ 475.740280] block drbd1: self > > 56E4CF14A115440C::02DCAB23D758DA48:02DBAB23D758DA49 > > bits:0 flags:0 > > [ 475.740288] block drbd1: peer > > F5A226CE3F2DA2F2::56E5CF14A115440D:56E4CF14A115440D > > bits:0 flags:0 > > [ 475.740295] block drbd1: uuid_compare()=-2 by rule 60 > > [ 475.740299] block drbd1: Writing the whole bitmap, full sync > required > > after drbd_sync_handshake. > > [ 475.757877] block drbd1: bitmap WRITE of 2400 pages took 4 jiffies > > [ 475.757888] block drbd1: 300 GB (78640791 bits) marked > out-of-sync by > > on disk bit-map. > > [ 475.758018] block drbd1: peer( Unknown -> Secondary ) conn( > > WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate ) > > [ 475.800134] block drbd1: receive bitmap stats [Bytes(packets)]: > plain > > 0(0), RLE 23(1), total 23; compression: 100.0% > > [ 475.802697] block drbd1: send bitmap stats [Bytes(packets)]: plain > > 0(0), RLE 23(1), total 23; compression: 100.0% > > [ 475.802717] block drbd1: conn( WFBitMapT -> WFSyncUUID ) > > [ 475.815155] block drbd1: updated sync uuid > > CEF5B26573C154CC::02DCAB23D758DA48:02DBAB23D758DA49 > > [ 475.815377] block drbd1: helper command: /sbin/drbdadm > > before-resync-target minor-1 > > [ 475.820270] block drbd1: helper command: /sbin/drbdadm > > before-resync-target minor-1 exit code 0 (0x0) > > [ 475.820293] block drbd1: conn( WFSyncUUID -> SyncTarget ) disk( > > Outdated -> Inconsistent ) > > [ 475.820306] block drbd1: Began resync as SyncTarget (will sync > > 314563164 KB [78640791 bits set]). > > [ 538.518371] block drbd1: peer( Secondary -> Primary ) > > [ 538.548954] block drbd1: role( Secondary -> Primary ) > > [ 2201.521232] block drbd1: conn( SyncTarget -> PausedSyncT ) > user_isp( > > 0 -> 1 ) > > [ 2201.521237] block drbd1: Resync suspended > > [ 2301.930484] block drbd1: conn( PausedSyncT -> SyncTarget ) > user_isp( > > 1 -> 0 ) > > [ 2301.930490] block drbd1: Syncer continues. > > [ 5216.750314] block drbd1: Resync done (total 4740 sec; paused > 100 sec; > > 67792 K/sec) > > [ 5216.750323] block drbd1: 98 % had equal
Re: [DRBD-user] drbd sycing again from the start
On 2019-10-15 4:58 p.m., Paras pradhan wrote: > Hi > > I have a two node drbd 8 cluster. We are doing some test and while drbd > resources are consistent/synced on both nodes, I powered off both nodes > and took a bare metal backup using clonezilla. > > Then I restored the both nodes using the backup and started drbd on both > nodes. On one of the nodes it starts to sync all over again. > > My question is: when I took the backup drbd resources are synced and why > it is starting all over again? I hope I explained clearly. > > Thanks in advance ! > Paras. Do you have the system logs from when you started DRBD on the nodes post-recovery? There should be DRBD log entries on both nodes as DRBD started. The reason/trigger of the resync will likely be explained in there. If not, please share the logs. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] How long does it take OOS to clear
Out of Sync is a question of what has changed locally versus what has reached the peer. It seems like you're generating changes on the Primary faster than the Secondary can receive them. So one answer is to speed up your replication link and/or the speed of the storage on the Secondary, depending on which is the bottle-neck. The other option is to switch to "Protocol C", which tells DRBD to not consider a write complete until it has reached both nodes. This will effectively slow down your Primary node's storage to whatever speed the Secondary can be written to, however, and may not be acceptable in your use case (see back to point 1 above). digimer On 2019-10-07 10:40 a.m., G C wrote: > I have an instance that seems to get OOS down lower and once in a while > it hits 0 but not very often. Typically my oos is about 18-20, > is there a way to clear this outside of the verify which takes a long > time? Is there something deeper I can dig into that would stop the oos > from occurring? > > Thank you > > > > ___ > Star us on GITHUB: https://github.com/LINBIT > drbd-user mailing list > drbd-user@lists.linbit.com > https://lists.linbit.com/mailman/listinfo/drbd-user > -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Occasional split-brain at boot time
On 2019-04-17 12:20 p.m., JCA wrote: I have a two-node cluster, in the way of two CentOS 7 VMs, as follows: Cluster name: ClusterOne Stack: corosync Current DC: two (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum Last updated: Wed Apr 17 09:43:42 2019 Last change: Wed Apr 17 09:39:52 2019 by root via cibadmin on one 2 nodes configured 4 resources configured Online: [ one two ] Full list of resources: MyAppCluster(ocf::myapps:MyApp):Started one Master/Slave Set: DrbdDataClone [DrbdData] Masters: [ one ] Slaves: [ two ] DrbdFS(ocf::heartbeat:Filesystem):Started one Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled The DRBD software that I using is the following: drbd84-utils.x86_64 9.6.0-1.el7.elrepo @elrepo kmod-drbd84.x86_64 8.4.11-1.1.el7_6.elrepo @elrepo The nodes have been configured to share an ext4 partition, DrbdFS has been configured to start before MyAppCluster, and the ClusterOne cluster has been configured to start automatically at boot time. The setup above works, in that node two takes over from node one when the latter becomes unreachable, and the DrbdFS filesystem automatically becomes available to node two, at the correct mount point, in that situation. Now when I reboot one and two, occasionally - but often enough to make me feel uneasy - DrbdFS comes up in a split-brain condition. What follows are the boot time syslog traces I typically get in such a case: Apr 17 09:35:59 one pengine[3663]: notice: * Start ClusterOne ( one ) Apr 17 09:35:59 one pengine[3663]: notice: * Start DrbdFS ( one ) Apr 17 09:35:59 one pengine[3663]: notice: Calculated transition 4, saving inputs in /var/lib/pacemaker/pengine/pe-input-560.bz2 Apr 17 09:35:59 one crmd[3664]: notice: Initiating monitor operation DrbdData_monitor_3 on two Apr 17 09:35:59 one crmd[3664]: notice: Initiating start operation DrbdFS_start_0 locally on one Apr 17 09:35:59 one kernel: drbd myapp-data: Handshake successful: Agreed network protocol version 101 Apr 17 09:35:59 one kernel: drbd myapp-data: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES. Apr 17 09:35:59 one kernel: drbd myapp-data: conn( WFConnection -> WFReportParams ) Apr 17 09:35:59 one kernel: drbd myapp-data: Starting ack_recv thread (from drbd_r_myapp-dat [4406]) Apr 17 09:35:59 one kernel: block drbd1: drbd_sync_handshake: Apr 17 09:35:59 one kernel: block drbd1: self 002DDA8B166FC899:8DD977B102052FD2: BE3891694D7BCD54:BE3791694D7BCD54 bits:0 flags:0 Apr 17 09:35:59 one kernel: block drbd1: peer D12D1947C4ECF940:8DD977B102052FD2: BE3891694D7BCD54:BE3791694D7BCD54 bits:32 flags:0 Apr 17 09:35:59 one kernel: block drbd1: uuid_compare()=100 by rule 90 Apr 17 09:35:59 one kernel: block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1 Apr 17 09:35:59 one Filesystem(DrbdFS)[4531]: INFO: Running start for /dev/drbd1 on /var/lib/myapp Apr 17 09:35:59 one kernel: block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1 exit code 0 (0x0) Apr 17 09:35:59 one kernel: block drbd1: Split-Brain detected but unresolved, dropping connection! Apr 17 09:35:59 one kernel: block drbd1: helper command: /sbin/drbdadm split-brain minor-1 Apr 17 09:35:59 one kernel: drbd myapp-data: meta connection shut down by peer. Apr 17 09:35:59 one kernel: drbd myapp-data: conn( WFReportParams -> NetworkFailure ) Apr 17 09:35:59 one kernel: drbd myapp-data: ack_receiver terminated Apr 17 09:35:59 one kernel: drbd myapp-data: Terminating drbd_a_myapp-dat Apr 17 09:35:59 one kernel: block drbd1: helper command: /sbin/drbdadm split-brain minor-1 exit code 0 (0x0) Fixing the problem is not difficult, by manual intervention, once both nodes are up and running. However, I would like to understand why the split-brain condition takes sometimes on booting up and, more importantly, how to prevent this from happening, if at all possible. Suggestions? Stonith in pacemaker, once tested, fencing in DRBD. This is what fencing is for. digimer ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] I need a little help with drbd and pacemaker - node never promoted
The thing is, even though it's a test system, pacemaker and DRBD will
still operate as if it is critical. Turning off stonith won't properly
emulate production because when a node enters an unknown state, the
system will no longer behave predictably.
digimer
On 2019-04-15 12:13 p.m., Graham Smith wrote:
Thanks for the reply
I'm just doing some basic testing and not using stonith, but I
understand why this is important.
I rebuilt my setup from scratch and it worked the 2nd time. Not sure
what I got wrong, but I now have the basics working.
--
best regards
Graham
-Original Message-
*From*: digimer <mailto:digimer%20%3cli...@alteeve.ca%3e>>
*To*: Graham Smith <mailto:graham%20smith%20%3cgps1...@gmail.com%3e>>,
drbd-user@lists.linbit.com <mailto:drbd-user@lists.linbit.com>
*Subject*: Re: [DRBD-user] I need a little help with drbd and
pacemaker - node never promoted
*Date*: Mon, 15 Apr 2019 11:24:36 -0400
For one; Enable and test stonith in Pacemaker. When a node can be
failed and fenced, then configure drbd to use fencing:
resource-and-stonith; and set the {un,}fence-handler to
crm-{un,}fence-peer.sh.
digimer
On 2019-04-12 11:51 p.m., Graham Smith wrote:
Hi
1st time user, trying to set up drbd with pacemaker for a potental
NFS server project. I've tried following 3 guides and I can not get
drbd and pacemaker to work together. And guidance is gratefully received.
Centos 7.5 on 2 nodes
uname -a
Linux node-1 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45
UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
I have drbd installed and initially it is syncing a disk between 2
nodes (prior to pcs cluster)
drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ d458166f5f4740625e5ff215f62366aca60ca37b\
build\ by\ mockbuild@\,\ 2018-11-03\ 14:14:44
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090010
DRBD_KERNEL_VERSION=9.0.16
DRBDADM_VERSION_CODE=0x090600
DRBDADM_VERSION=9.6.0
drbdadm status
drbd0 role:Primary
disk:UpToDate
node-2 role:Secondary
peer-disk:UpToDate
So far so good
cat /proc/drbd
version: 9.0.16-1 (api:2/proto:86-114)
GIT-hash: ab9777dfeaf9d619acc9a5201bfcae8103e9529c build by
mockbuild@, 2018-11-03 13:54:24
Transports (api:16): tcp (9.0.16-1)
# the output is different than I see in online guides (maybe
different in v9?).
https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html/Clusters_from_Scratch/_initialize_drbd.html
-
[
root@pcmk-
<mailto:root@pcmk->
1 ~]# cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@,
2018-04-26 12:10:42
1: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C rs
ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:524236
When I create a PCS cluster and add drbd as a resource, I do not get
a node to become a master
pcs -f drbd_cfg resource create NFSData ocf:linbit:drbd
drbd_resource=drbd0 op monitor interval=60
pcs -f drbd_cfg resource master NFSDataClone NFSData master-max=1
master-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs status
Cluster name: mycluster
Stack: corosync
Current DC: node-1 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition
with quorum
Last updated: Fri Apr 12 23:43:36 2019
Last change: Fri Apr 12 23:43:32 2019 by root via cibadmin on node-1
2 nodes configured
2 resources configured
Online: [ node-1 node-2 ]
Full list of resources:
Master/Slave Set: NFSDataClone [NFSData]
Slaves: [ node-1 node-2 ]
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
Here is the output of pcs config
pcs config
Cluster Name: mycluster
Corosync Nodes:
node-1 node-2
Pacemaker Nodes:
node-1 node-2
Resources:
Master: NFSDataClone
Meta Attrs: master-node-max=1 clone-max=2 notify=true master-max=1
clone-node-max=1
Resource: NFSData (class=ocf provider=linbit type=drbd)
Attributes: drbd_resource=drbd0
Operations: demote interval=0s timeout=90 (NFSData-demote-interval-0s)
monitor interval=60s (NFSData-monitor-interval-60s)
notify interval=0s timeout=90 (NFSData-notify-interval-0s)
promote interval=0s timeout=90
(NFSData-promote-interval-0s)
reload interval=0s timeout=30 (NFSData-reload-interval-0s)
start interval=0s timeout=240 (NFSData-start-interval-0s)
stop interval=0s timeout=100 (NFSData-stop-interval-0s)
Stonith Devices:
Fencing Levels:
Location Constraints:
Ordering Constraints:
Colocation Constraints:
Ticket Constraints:
Alerts:
No alerts defined
Resources Defaults:
resource-stickiness: 100
Operations Defaults:
No defaults set
Cluster Properties:
cluster-infrastructure: corosync
cluster-name: mycluster
dc-version: 1.1.19-8.el7_6.4-c3c624ea3d
default-resource-stickiness: INFINITY
have-watchdog: false
no-quorum-policy: ignore
stonith-enabled: false
Quorum:
Options:
--
Re: [DRBD-user] I need a little help with drbd and pacemaker - node never promoted
For one; Enable and test stonith in Pacemaker. When a node can be failed
and fenced, then configure drbd to use fencing: resource-and-stonith;
and set the {un,}fence-handler to crm-{un,}fence-peer.sh.
digimer
On 2019-04-12 11:51 p.m., Graham Smith wrote:
Hi
1st time user, trying to set up drbd with pacemaker for a potental NFS
server project. I've tried following 3 guides and I can not get drbd
and pacemaker to work together. And guidance is gratefully received.
Centos 7.5 on 2 nodes
uname -a
Linux node-1 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC
2019 x86_64 x86_64 x86_64 GNU/Linux
I have drbd installed and initially it is syncing a disk between 2
nodes (prior to pcs cluster)
drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ d458166f5f4740625e5ff215f62366aca60ca37b\
build\ by\ mockbuild@\,\ 2018-11-03\ 14:14:44
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090010
DRBD_KERNEL_VERSION=9.0.16
DRBDADM_VERSION_CODE=0x090600
DRBDADM_VERSION=9.6.0
drbdadm status
drbd0 role:Primary
disk:UpToDate
node-2 role:Secondary
peer-disk:UpToDate
So far so good
cat /proc/drbd
version: 9.0.16-1 (api:2/proto:86-114)
GIT-hash: ab9777dfeaf9d619acc9a5201bfcae8103e9529c build by
mockbuild@, 2018-11-03 13:54:24
Transports (api:16): tcp (9.0.16-1)
# the output is different than I see in online guides (maybe different
in v9?).
https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html/Clusters_from_Scratch/_initialize_drbd.html
-
[root@pcmk- <mailto:root@pcmk->1 ~]# cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@,
2018-04-26 12:10:42
1: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C rs
ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:524236
When I create a PCS cluster and add drbd as a resource, I do not get a
node to become a master
pcs -f drbd_cfg resource create NFSData ocf:linbit:drbd
drbd_resource=drbd0 op monitor interval=60
pcs -f drbd_cfg resource master NFSDataClone NFSData master-max=1
master-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs status
Cluster name: mycluster
Stack: corosync
Current DC: node-1 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition
with quorum
Last updated: Fri Apr 12 23:43:36 2019
Last change: Fri Apr 12 23:43:32 2019 by root via cibadmin on node-1
2 nodes configured
2 resources configured
Online: [ node-1 node-2 ]
Full list of resources:
Master/Slave Set: NFSDataClone [NFSData]
Slaves: [ node-1 node-2 ]
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
Here is the output of pcs config
pcs config
Cluster Name: mycluster
Corosync Nodes:
node-1 node-2
Pacemaker Nodes:
node-1 node-2
Resources:
Master: NFSDataClone
Meta Attrs: master-node-max=1 clone-max=2 notify=true master-max=1
clone-node-max=1
Resource: NFSData (class=ocf provider=linbit type=drbd)
Attributes: drbd_resource=drbd0
Operations: demote interval=0s timeout=90 (NFSData-demote-interval-0s)
monitor interval=60s (NFSData-monitor-interval-60s)
notify interval=0s timeout=90 (NFSData-notify-interval-0s)
promote interval=0s timeout=90
(NFSData-promote-interval-0s)
reload interval=0s timeout=30 (NFSData-reload-interval-0s)
start interval=0s timeout=240 (NFSData-start-interval-0s)
stop interval=0s timeout=100 (NFSData-stop-interval-0s)
Stonith Devices:
Fencing Levels:
Location Constraints:
Ordering Constraints:
Colocation Constraints:
Ticket Constraints:
Alerts:
No alerts defined
Resources Defaults:
resource-stickiness: 100
Operations Defaults:
No defaults set
Cluster Properties:
cluster-infrastructure: corosync
cluster-name: mycluster
dc-version: 1.1.19-8.el7_6.4-c3c624ea3d
default-resource-stickiness: INFINITY
have-watchdog: false
no-quorum-policy: ignore
stonith-enabled: false
Quorum:
Options:
--
After creating the PCS cluster drbd now has no primary
drbdadm status
drbd0 role:Secondary
disk:UpToDate
node-2 role:Secondary
peer-disk:UpToDate
I'm stumped on how to get this to work, are there better guides, is
this a know bug. Thanks in advance for any guidance.
--
best regards
Graham
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Number of nodes supported for DRBD
On 2019-02-20 4:05 a.m., Malhar vora wrote: > Hi Experts, > > Does anyone know how many nodes can be supported for DRBD ? > > Can DRBD support count like 1000+ nodes ? 16 nodes. https://docs.linbit.com/docs/users-guide-9.0/#ch-configure Subsection 9.1.4; For (the current) maximum of 16 nodes there’ll be 120 host pairs to connect. ==== -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
[DRBD-user] Building DRBD9 on RHEL8
Hi all,
I asked this on the IRC channel, but I think it might be more useful
as a question that will get into the archives.
I'm trying to build DRBD9 (kernel module) on RHEL8, and I've hit a few
snags. I updated the .spec to pull out some SUSE checks (to get things a
bit simpler while I debug) and made a couple changes with macros. This
gets be further in the build, but it's currently failing with:
Processing files: kmod-drbd-9.0.16-1.x86_64
Processing files: drbd-kernel-debugsource-9.0.16-1.x86_64
error: Empty %files file
/root/rpmbuild/BUILD/drbd-9.0.16-1/debugsourcefiles.list
RPM build errors:
Macro %kmp_release defined but not used within scope
Macro %kmp_version defined but not used within scope
Macro %latest_kernel defined but not used within scope
Empty %files file
/root/rpmbuild/BUILD/drbd-9.0.16-1/debugsourcefiles.list
Here are all the details;
* Currently on git f1dc5977bf81a2aaf5244166840ea37da1593bbf
* Current .spec:
https://paste.fedoraproject.org/paste/8XbIXT9bdYc-~-Tvi5Ya-A/raw
* My .spec diff'ed against the generated one:
https://paste.fedoraproject.org/paste/1DgepPwkQnS9xXGo0GPZYw/raw
* Full build output:
https://paste.fedoraproject.org/paste/VPqvEwCKUQdhtpD8G1-Wkw/raw
* List of files in the build directory with sizes and modes:
https://paste.fedoraproject.org/paste/tFDKKwROJWdAbuidmUhQ~g/raw
I'm building as root at this stage just to take possible permission
errors out. Of course, once I have a working src.rpm, I'll rebuild under
an unprivileged user.
Below I will include the contents of the above links, just in case the
links disappear someday.
Thanks for any insight or help!
Digimer
* Current .spec:
Name: drbd-kernel
Summary: Kernel driver for DRBD
Version: 9.0.16
Release: 1
# always require a suitable userland
Requires: drbd-utils >= 9.2.0
%global tarball_version %(echo "%{version}-%{release}")
Source: http://oss.linbit.com/drbd/drbd-%{tarball_version}.tar.gz
License: GPLv2+
URL: http://www.drbd.org/
BuildRequires: redhat-rpm-config
BuildRequires: kernel-devel
%description
This module is the kernel-dependent driver for DRBD. This is split out so
that multiple kernel driver versions can be installed, one for each
installed kernel.
%prep
%setup -q -n drbd-%{tarball_version}
# Concept stolen from sles kernel-module-subpackage:
# include the kernel version in the package version,
# so we can have more than one kmod-drbd.
# Needed, because even though kABI is still "compatible" in RHEL 6.0 to 6.1,
# the actual functionality differs very much: 6.1 does no longer do
BARRIERS,
# but wants FLUSH/FUA instead.
# For convenience, we want both 6.0 and 6.1 in the same repository,
# and have yum/rpm figure out via dependencies, which kmod version
should be installed.
# This is a dirty hack, non generic, and should probably be enclosed in
some "if-on-rhel6".
%define kernel_version %(uname -r | sed -r 's/\.x86_64//')
%define kdir /usr/src/kernels/%(uname -r)
%define _this_kmp_version %{version}_%{kernel_version}
%kernel_module_package -v %_this_kmp_version -n drbd -f filelist-redhat
%build
rm -rf obj
mkdir obj
ln -s ../drbd-headers obj/
for flavor in %flavors_to_build; do
cp -r drbd obj/$flavor
make -C obj/$flavor %{_smp_mflags} all KDIR=%{kdir $flavor}
done
%install
export INSTALL_MOD_PATH=$RPM_BUILD_ROOT
%if %{defined kernel_module_package_moddir}
export INSTALL_MOD_DIR=%{kernel_module_package_moddir drbd}
%else
export INSTALL_MOD_DIR=extra/drbd
%endif
# Very likely kernel_module_package_moddir did ignore the parameter,
# so we just append it here. The weak-modules magic expects that location.
[ $INSTALL_MOD_DIR = extra ] && INSTALL_MOD_DIR=extra/drbd
for flavor in %flavors_to_build ; do
make -C %{kernel_source $flavor} modules_install \
M=$PWD/obj/$flavor
kernelrelease=$(cat %{kernel_source
$flavor}/include/config/kernel.release || make -s -C %{kernel_source
$flavor} kernelrelease)
mv obj/$flavor/.kernel.config.gz obj/k-config-$kernelrelease.gz
mv obj/$flavor/Module.symvers
../../RPMS/Module.symvers.$kernelrelease.$flavor.%{_arch}
done
mkdir -p $RPM_BUILD_ROOT/etc/depmod.d
echo "override drbd * weak-updates" \
> $RPM_BUILD_ROOT/etc/depmod.d/drbd.conf
%clean
rm -rf %{buildroot}
%changelog
* Thu Oct 25 2018 Philipp Reisner - 9.0.16-1
- New upstream release.
* Tue Aug 14 2018 Philipp Reisner - 9.0.15-1
- New upstream release.
* Tue May 01 2018 Lars Ellenberg - 9.0.14-1
- New upstream release.
* Tue Apr 17 2018 Philipp Reisner - 9.0.13-1
- New upstream release.
* Mon Jan 22 2018 Philipp Reisner - 9.0.12-1
- New upstream release.
* Tue Jan 09 2018 Roland Kammerer - 9.0.11-1
- New upstream release.
* Fri Dec 22 2017 Roland Kammerer - 9.0.10-1
- New upstream release.
* Thu Aug 31 2017 Philipp Reisner - 9.0.9-1
- New upstream release.
* Mon Jun 19 2017 Philipp Reisner - 9.0.8-1
- New upstream release.
* Fri Mar 31 2017 Philipp Re
Re: [DRBD-user] Ask some questions about the splitting of the dual master device
On 2018-12-09 8:28 p.m., Su Hua wrote: > Excuse me, the latest drbd driver can provide the arbitration service in > the case of split-brain？instead of using the script that is fixed > afterwards, which may cause partial data loss under the GFS2 cluster > file system. DRBD needs to be fenced in a way that informs DLM that the lost node has been fenced. I do not know of any fence handlers that do this is a pure DRBD install. So for practical purposes, no, you will need pacemaker with a proper stonith configuration to avoid split-brains in the first place. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Configuring a two-node cluster with redundant nics on each node?
On 2018-10-19 11:16 a.m., Bryan K. Walton wrote: On Thu, Oct 18, 2018 at 04:47:53PM +1100, Adi Pircalabu wrote: Why aren't you using Ethernet bonding? Thanks Adi, We are rethinking our network configuration. We may do our replication through a directly cabled and bonded connection, and bypass our switches. This would simplify our drbd configuration. We've used mode=1 (active-passive) bonding under DRBD for years across numerous installs and countless test and prod failures without issue. We still run through switches (link1 to switch 1, link2 to switch 2, hitless failover config on the stack). We can and have failed NICs, cables and switches without interruption. We've documented this setup here; https://www.alteeve.com/w/Build_an_m2_Anvil!#Logical_Map.3B_Hardware_And_Plumbing digimer ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] split brain on both nodes
On 2018-10-18 4:00 p.m., Lars Ellenberg wrote: > On Wed, Oct 17, 2018 at 03:11:42PM -0400, Digimer wrote: >> On 2018-10-17 5:35 a.m., Adam Weremczuk wrote: >>> Hi all, >>> >>> Yesterday I rebooted both nodes a couple of times (replacing BBU RAID >>> batteries) and ended up with: >>> >>> drbd0: Split-Brain detected but unresolved, dropping connection! >> >> Fencing prevents this. >> >>> on both. >>> >>> node1: /drbd-overview// >>> //0:r0/0 StandAlone Primary/Unknown UpToDate/DUnknown /srv/test1 ext4 >>> 3.6T 75G 3.4T 3% / >>> >>> node2: /drbd-overview // >>> //0:r0/0 StandAlone Secondary/Unknown UpToDate/DUnknown/ >>> >>> I understand there is a good chance (but not absolute guarantee) that >>> node1 holds consistent and up to date data. >>> >>> Q1: >>> >>> Is it reasonably possible to mount /dev/drbd0 (/dev/sdb1) on node2 in >>> read only mode? >>> >>> I would like to examine the data before discarding and syncing >>> everything from node1. >> >> Yes. You can also promote node 2 to examine it as well. >> >>> /drbdadm disconnect all// >>> //drbdadm -- --discard-my-data connect all/ >> >> Discarding the data will trigger a resync and resolve the split-brain, >> but of course, any changes on the discarded node will be lost. >> >>> Q2: >>> >>> Will the above completely purge all data on node2 or just drbd metadata? >>> >>> I.e. will all 75G have to be fully copied block by block or a lot less? >> >> It will do a full resync. > > Even after a "split brain" (rather: data divergence), > DRBD usually can do an "incremental, bitmap based" resync: > DRBD knows which blocks have changed on each node, and will sync the > blocks described by the bit-or of those out-of-sync bitmaps. Glad to hear I was wrong on that, cool. >>> I'm concerned about time and impact on performance when it comes to >>> terabytes of data. >>> >>> Regards, >>> Adam >> >> The resync (on 8.4) adapts the resync rate to minimize impact on >> applications using the storage. As it slows itself down to "stay out of >> the way", the resync time increases of course. You won't have redundancy >> until the resync completes. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] split brain on both nodes
On 2018-10-17 5:35 a.m., Adam Weremczuk wrote: > Hi all, > > Yesterday I rebooted both nodes a couple of times (replacing BBU RAID > batteries) and ended up with: > > drbd0: Split-Brain detected but unresolved, dropping connection! Fencing prevents this. > on both. > > node1: /drbd-overview// > //0:r0/0 StandAlone Primary/Unknown UpToDate/DUnknown /srv/test1 ext4 > 3.6T 75G 3.4T 3% / > > node2: /drbd-overview // > //0:r0/0 StandAlone Secondary/Unknown UpToDate/DUnknown/ > > I understand there is a good chance (but not absolute guarantee) that > node1 holds consistent and up to date data. > > Q1: > > Is it reasonably possible to mount /dev/drbd0 (/dev/sdb1) on node2 in > read only mode? > > I would like to examine the data before discarding and syncing > everything from node1. Yes. You can also promote node 2 to examine it as well. > /drbdadm disconnect all// > //drbdadm -- --discard-my-data connect all/ Discarding the data will trigger a resync and resolve the split-brain, but of course, any changes on the discarded node will be lost. > Q2: > > Will the above completely purge all data on node2 or just drbd metadata? > > I.e. will all 75G have to be fully copied block by block or a lot less? It will do a full resync. > I'm concerned about time and impact on performance when it comes to > terabytes of data. > > Regards, > Adam The resync (on 8.4) adapts the resync rate to minimize impact on applications using the storage. As it slows itself down to "stay out of the way", the resync time increases of course. You won't have redundancy until the resync completes. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] drbd resyncing entire device after each reboot
00:B6D88D552E97D8B6:B6D78D552E97D8B7 > Oct 5 21:36:47 claire kernel: [ 384.275945] block drbd0: helper command: > /sbin/drbdadm before-resync-target minor-0 > Oct 5 21:36:47 claire kernel: [ 384.279872] block drbd0: helper command: > /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0) > Oct 5 21:36:47 claire kernel: [ 384.279905] block drbd0: conn( WFSyncUUID > -> SyncTarget ) disk( Outdated -> Inconsistent ) > Oct 5 21:36:47 claire kernel: [ 384.279949] block drbd0: Began resync as > SyncTarget (will sync 26842726364 KB [6710681591 bits set]). > > Probably the explanation is simple, I just do not see it. > > If you need the configuration (although it should be identical to > similar drbd configs which are working without problems) I am happy to > provide it. > > Best and many thanks if any body could shed some light on this, > Hp Can you share your config? Are you using thin LVM? Also, 8.4.7 is _ancient_. Nearly countless bug fixes since then, which may or may not relate. In any case, updating is _strongly_ recommended. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Node1 Standalone Inconsistent/Outdated Node 2 WFConnection Inconsisten/DUnknown - how to recover?
On 2018-10-02 10:12 AM, Ron wrote: > Hi, > > I've inherited a setup that has been running for a very long time, > but this morning both were down, and showed: > > [root@node-01 ~]# drbd-overview > 0:drbd0 StandAlone Secondary/Unknown Inconsistent/Outdated r- > 1:drbd1 Connected Secondary/Secondary UpToDate/UpToDate C > r- > > [root@node-02 ~]# drbd-overview > 0:drbd0 WFConnection Secondary/Unknown Inconsistent/DUnknown C r- > 1:drbd1 Connected Secondary/Secondary UpToDate/UpToDate C r- > > We had a power failure two weeks ago, but nothing seemed wrong. > > How do I recover from this situation? How can I find out which of the > nodes has > the most recent information? And then mark it UpToDate? > > Thanks, > Ron Tail the system logs on both nodes, then on node 1, run 'drbdadm connect r0'. Wait for things to settle, then share the new status of /proc/drbd and the log output from the two nodes please. digimer -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Mount and use disk while Inconsistent?
Yup, its fine. Note though; If the UpToDate node goes offline, the Inconsistent node will force itself to Secondary and be unusable. So while it's possible to mount and use, be careful that whatever is being used can handle having the storage ripped out from under it. digimer On 2018-09-21 01:52 PM, Dan Ragle wrote: Just double checking. Is it ok to have a dual-primary setup where both nodes are primary while one is still syncing? [node1]# drbdadm status r0 role:Primary volume:0 disk:UpToDate volume:1 disk:UpToDate node2.mydomain.com role:Primary volume:0 replication:SyncSource peer-disk:Inconsistent done:99.46 volume:1 peer-disk:UpToDate First time I've seen it in my testing. Nothing complained about it so I *think* it's ok ... ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Any way to jump over initial sync ?
On 2018-08-28 12:59 PM, Julien Escario wrote: Hello, Just wanted to know : is there a way to get rid of initial sync with linstor and zfs backend ? Right now, I have a 1TB volume to create and initial sync is vry long. I think it's mostly due to unavailability of thinly provisionned ZFS resources but perhaps is there a way to suspend resync and ask the system to simply consider both resources as sync'ed ? Best regards, Julien In 8.4, you could do 'drbdadm new-current-uuid --clear-bitmap '. Be aware though; The bits won't match until/unless a given block is written. So a 'verify' will be problematic. digimer ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] drbd-9.0.15
On 2018-08-14 08:36 AM, Philipp Reisner wrote: > Hi, > > This is an upgrade ever drbd-9 user should follow. It has two important > fixes in the areas of > > * handling IO errors reported by the backing device > > handling of IO errors on the backend was completely broken since > drbd-9.0 including the recovery options like replacing a failed > disk. When the disk was replaced even worse it was possible that > DRBD would read from the new disk before the full sync finished. > All fixed now, but very embarrassing. > > * correctly handle UUIDs in case of live-migration > > That was the root cause for various strange behavior. E.g. a node > considering some other as not up-to-date while the peer considers > itself as up-to-date. > > The goody of the release is that the submit code path was optimized > a bit, and that gives up to 30% increase (depending on CPU model and > performance of the backing device) in IOPs. > > A lot of effort was spend to write more tests for the drbd9 test suite > ( https://github.com/LINBIT/drbd9-tests ). DRBD-8.4 had its own, which > was more complete at its time, but now it is overdue to have a testing > coverage at least as good for the drbd-9 code base. > > Apart form wok on the testsuite we will continue to put effort into > optimizing the IO submit code path. Very fast NVMe devices keep the > pressure on us to be able to fully utilize them when used as backing > device for DRBD. > > Note: We will update the PPA on Thursday (Aug 16). Sorry for the delay > (vacations and a bank holiday are the reasons) > > 9.0.15 (api:genl2/proto:86-114/transport:14) > > * fix tracking of changes (on a secondary) against the lost disk of a >primary and also fix re-attaching in case the disk is replaced (has >new meta-data) > * fix live migrate of VMs on DRBD when migrated to/from diskless >nodes; before that fix a race condition can lead to one of the nodes >seeing the other one as consistent only > * fix an IO deadlock in DRBD when the activity log on a secondary runs full; >In the real world, this was very seldom triggered but can be easily >reproduced with a workload that touches one block every 4M and writes >them all in a burst > * fix hanging demote after IO error followed by attaching the disk again >and the corresponding resync > * fix DRBD dropping connection after an IO error on the secondary node > * new module parameter to disable support for older protocol versions, >an in case you configured peers that are not expected to connect it >might have positive effects because then this node does not need to >assume that such peer is ancient > * improve details when online changing devices from diskless to with disk and >vice versa. (Including peers freeing bitmap slots) > * remove no longer relevant compat tests > * expose openers via debugfs; that helps to answer the question why does >DRBD not demote to secondary, why does it give tell me "Device is held >open by someone" > * optimize IO submit code path; this can improve IOPs up to 30% on a system >with fast backend storage; lowers CPU load caused by DRBD on every workload > * compat for v4.18 kernel > > http://www.linbit.com/downloads/drbd/9.0/drbd-9.0.15-1.tar.gz > https://github.com/LINBIT/drbd-9.0/releases/tag/drbd-9.0.15 > > best regards, > Phil Congrats on the release! -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] frequent split brains without network interruption
Fencing prevents split-brains, period. You absolutely want to set that up. On 2018-06-12 04:02 AM, Christian Still wrote: > Hello Roland, Hello Digimer, > > thank you for your fast response, > > at the moment I have upgraded one host from source to: > > cat /proc/drbd > version: 8.4.11-1 (api:1/proto:86-101) > > the second one will be upgraded today in the afternoon, > I will keep you informed, if the problem is solved with the new version. > > Actually there is no fencing in use. > > Thanks again and best regards > > SC > > -Ursprüngliche Nachricht- > Von: drbd-user-boun...@lists.linbit.com > [mailto:drbd-user-boun...@lists.linbit.com] Im Auftrag von Digimer > Gesendet: Dienstag, 12. Juni 2018 09:06 > An: Roland Kammerer ; drbd-user@lists.linbit.com > Betreff: Re: [DRBD-user] frequent split brains without network interruption > > On 2018-06-11 04:31 AM, Roland Kammerer wrote: >> On Fri, Jun 08, 2018 at 08:56:53AM +, Christian Still wrote: >>> Hi all, >>> >>> version: 8.4.5 (api:1/proto:86-101) >> >> Hi, >> >> we are at 8.4.11 [1], please upgrade and check if it still occurs. >> >> Regards, rck > > Also, are you using fencing? > > > -- > Digimer > Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested > in the weight and convolutions of Einstein’s brain than in the near certainty > that people of equal talent have lived and died in cotton fields and > sweatshops." - Stephen Jay Gould > ___ > drbd-user mailing list > drbd-user@lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user > -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] frequent split brains without network interruption
On 2018-06-11 04:31 AM, Roland Kammerer wrote: > On Fri, Jun 08, 2018 at 08:56:53AM +, Christian Still wrote: >> Hi all, >> >> version: 8.4.5 (api:1/proto:86-101) > > Hi, > > we are at 8.4.11 [1], please upgrade and check if it still occurs. > > Regards, rck Also, are you using fencing? -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] About outdated and inconsistent state
On 2018-06-05 04:23 AM, Roland Kammerer wrote: > On Tue, Jun 05, 2018 at 01:07:48PM +0800, 唐宇奕 wrote: >> Thanks for reply, my situation is we want to backup a git server >> repository, is drbd enough? > > Hm, DRBD is not really a "backup solution", it perfectly replicates your > mistakes live and in colors... But let's not be picky about wording. > > Putting my DRBD hat off, just pushing to a second git remote > (manually/automatically) would work as well. Git obviously is good in > being distributed... > > Regards, rck Agreed; DRBD is a replication technology meant to protect against hardware faults. Think of it like RAID level 1, but for a full machine. You wouldn't use RAID 1 for backups. So DRBD is very useful, but it scratches a different itch. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] About outdated and inconsistent state
On 2018-05-31 10:33 PM, 唐宇奕 wrote: Hello, I wonder when exactly can outdated and inconsistent state of data can happen and how often does that happen? And is there a way to use inconsistent or outdated data after all if the primary node fails? The full answer is here; https://docs.linbit.com/docs/users-guide-8.4/#s-disk-states The short answer is; Outdated means the data is consistent, but old. Using it will mean any data written after the disk was disconnected will be lost. This happens when a node is cleanly disconnected while another node is still Primary. Inconsistent means that the data on the disk had started to sync from an UpToDate node, but that the resync has not finished. The data on the disk is likely not usable (specifically, it might be a mix of new and old data). digimer ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] about split-brain
On 2018-04-20 09:35 AM, Lars Ellenberg wrote: > On Thu, Apr 19, 2018 at 05:06:03PM +0300, Veli Cakmak wrote: >> hello guys, i have 2 drbd node (primary/secondary) I try to solve split >> brain without any lost data. How can i find last updated node ? Is there >> any command or something like ? > > Divergence of instances of supposedly replicated data is > typically the result of a cluster split brain, > thus "split-brain" in this context is often used as a synonym > for data divergence. > > You *cannot* resolve data divergence without data loss. > There is no "merge". Not on the block device level. > > If you wanted to "merge" both sets of changes, > you could keep them both, > bring them "online" in "single-user" mode > (whatever that means for the data set in question) > and "merge" changes (files, database records, ...) > on the application level. > > That may be trivial (e.g. monitoring time series in rrd files), > though still a lot of effort. > Or it may take more time than you think, > cause a lot of frustration, > and finally result in the insight that merging the changes > is not possible in any sane way anyways. > > On the block level, > you have to throw away the changes on the "victim" instance(s), > and override with the changes of some other instance > which you declare to be the "winner". > > *automating* that is automating data loss > (throwing away the changes of the victim(s)). > > Again: > > Using DRBD after-split-brain auto recovery strategies > is configuring automatic data loss. > > This is for people who care more about being online > with "some" data than for being online with the "right" data. > > People who care about the "right" data > try to *avoid* data divergence in the first place. > > Which usually is achieved by a combination of out-of-band > communication of the DRBD instances (drbd resource fencing policies > and the fence-peer handler) and fencing on the cluster manager level. > > To reliably avoid "split brain" (data divergence), you need fencing > properly configured on both the pacemaker level and the DRBD level. > > If you rather want to "get away" without fencing, and instead > try to automate "recovery" from "split brain" situations, > you are automating data loss. > > Don't blame DRBD, it is just that flexible. > > The choice is yours. To elaborate on Lars' comments; USE FENCING! That is, a node should be power cycled (or at least disconnected from the network at the switch level) when it stops responding. If both nodes are alive, you can either let the faster one win, or, configure 'delay="15"' on the stonith config in pacemaker's fence config for the primary node so that the primary node gets a head start in shooting the secondary node. Automatic split-brain recovery is a terrible idea, for all the reasons Lars pointed out. Avoiding a split-brain in the first place with a proper stonith config is, by far, the way to go. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Interesting issue with drbd 9 and fencing
On 2018-02-13 09:39 AM, Lars Ellenberg wrote: > On Sun, Feb 11, 2018 at 02:43:27AM -0500, Digimer wrote: >> On 2018-02-11 01:42 AM, Digimer wrote: >> >> >> Hi all, >> >> I've setup a 3-node cluster (config below). Basically, Node 1 & 2 are >> protocol C and have resource-and-stonith fencing. Node 1 -> 3 and 2 -> 3 are >> protocol A and fencing is 'dont-care' (it's >> not part of the cluster and would only ever be promoted manually). >> >> When I crash node 2 via 'echo c > /proc/sysrq-trigger', pacemaker >> detected the faults and so does DRBD. DRBD invokes the fence-handler as >> expected and all is good. However, I want to test >> breaking just DRBD, so on node 2 I used 'iptables -I INPUT -p tcp -m tcp >> --dport 7788:7790 -j DROP' to interrupt DRBD traffic. When this is done, the >> fence handler is not invoked. > > iptables command may need to be changed, to also drop --sport, > and for good measure, add the same to the OUTPUT chain. > DRBD connections (are can be) established in both directions. > You blocked only one direction. > > Maybe do it more like this: > iptables -X drbd > iptables -N drbd > iptables -I INPUT -p tcp --dport 7788:7790 -j drbd > iptables -I INPUT -p tcp --sport 7788:7790 -j drbd > iptables -I OUTPUT -p tcp --dport 7788:7790 -j drbd > iptables -I OUTPUT -p tcp --sport 7788:7790 -j drbd > > Then toggle: > break: iptables -I drbd -j DROP > heal: iptables -F drbd > > (beware of typos, I just typed this directly into the email) > >> Issue the iptables command on node 2. Journald output; >> >> >> -- Logs begin at Sat 2018-02-10 17:51:59 GMT. -- >> Feb 11 06:20:18 m3-a02n01.alteeve.com crmd[2817]: notice: State >> transition S_TRANSITION_ENGINE -> S_IDLE >> Feb 11 06:28:57 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0 >> m3-a02n02.alteeve.com: PingAck did not arrive in time. >> Feb 11 06:28:57 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0: susp-io( >> no -> fencing) > > Goes for suspend-io due to fencing policies, as configured. > >> Feb 11 06:28:57 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0 >> m3-a02dr01.alteeve.com: Preparing remote state change 1400759070 >> (primary_nodes=1, weak_nodes=FFFA) >> Feb 11 06:28:57 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0 >> m3-a02dr01.alteeve.com: Committing remote state change 1400759070 >> Feb 11 06:28:57 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0/0 drbd0 >> m3-a02n02.alteeve.com: pdsk( DUnknown -> Outdated ) > > But state changes are relayed through all connected nodes, > and node02 confirms that it now knows it is Outdated. > >> Feb 11 06:28:57 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0/0 drbd0: >> new current UUID: 769A55B47EB143CD weak: FFFA >> Feb 11 06:28:57 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0: susp-io( >> fencing -> no) > > Which means we may resume-io after bumping the data generation uuid tag, > and don't have to call out to any additional handler "helper" scripts. > >> Feb 11 06:28:57 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0 >> m3-a02n02.alteeve.com: conn( Unconnected -> Connecting ) >> Feb 11 06:29:18 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0 >> m3-a02n02.alteeve.com: Handshake to peer 1 successful: Agreed network >> protocol version 112 > > And since you only blocked one direction, > we can establish a new one anyways in the other direction. > > we do a micro (in this case even: empty) resync: > >> Feb 11 06:29:18 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0/0 drbd0 >> m3-a02n02.alteeve.com: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> >> SyncSource ) >> Feb 11 06:29:18 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0/0 drbd0 >> m3-a02n02.alteeve.com: Began resync as SyncSource (will sync 0 KB [0 bits >> set]). >> Feb 11 06:29:18 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0/0 drbd0 >> m3-a02n02.alteeve.com: updated UUIDs >> 769A55B47EB143CD::4CF0E17ADD9D1E0E:4161585F99D3837C >> Feb 11 06:29:18 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0/0 drbd0 >> m3-a02n02.alteeve.com: Resync done (total 1 sec; paused 0 sec; 0 K/sec) >> Feb 11 06:29:18 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0/0 drbd0 >> m3-a02n02.alteeve.com: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> >> Established ) >> Feb 11 06:29:18 m3-a02n01.alteeve.com kernel: drbd srv01-c7_0 >> m3-a02n02.alteeve.com: helper command: /sbin/drbdadm unfe
[DRBD-user] Managing a 3-node DRBD 9 resource in pacemker
Hi all, I was trying to manage DRBD in pacemaker for a test cluster I built. It's a 3-node cluster; Node 1 & 2 are in pacemaker and are protocol C with proper fencing. The third node is a DR node, protocol A. At first, I just added DRBD as a systemd service (quick and dirty, I know it's not good long term). When I had the DR node's drbd service running, everything was fine. When I down'ed it though, pacemaker said it timed out starting drbd (I assume because both nodes were still waiting for the DR node to connect). So I decided to try to switch to ocf:linbut:drbd, but I don't know how to set it up such that it doesn't care about the DR node's state (specially because the DR is not part of the pacemaker cluster). Is it possible with the current RA to setup such that it only monitors the two nodes in pacemaker and ignores the state of the DR node? Cheers -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
[DRBD-user] Interesting issue with drbd 9 and fencing
tbeat:VirtualDomain): Started m3-a02n01.alteeve.com Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled To verify, I can't connect to node 2; [root@m3-a02n01 ~]# telnet m3-a02n02.sn 7788 Trying 10.41.20.2... telnet: connect to address 10.41.20.2: Connection timed out Did it somehow maintain connection through node 3? If not, then a) Why didn't the fence-handler get invoked? b) Why is it still showing connected? If so, then is the connection between node 1 and 2 still protocol C, even if the connection between 1 <-> 3 and 2 <-> 3 are protocol A? Thanks! -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Moving TeraBytes to drbd device
Good fencing prevents split-brains, period. Setup pacemaker, setup and
test stonith, configure DRBD to use 'fencing resource-and-stonith;' and
enable the crm-{un,}fence-peer.sh' {un,}fence-handlers. The setup
pacemaker to colocate the DRBD resource -> FS -> IP.
digimer
On 2017-10-05 05:01 PM, José Andrés Matamoros Guevara wrote:
> Don't know yet. Thinking about using pacemaker, for high availability. But
> have a manually recovery to avoid split brain - corruption data situations.
> Maybe we are going to use three-four servers to have a disaster recovery site.
>
> Andres.
>
> -Original Message-
> From: Digimer [mailto:li...@alteeve.ca]
> Sent: Thursday, 5 October, 2017 12:43 PM
> To: José Andrés Matamoros Guevara <amatamo...@ie.com.sv>;
> drbd-user@lists.linbit.com
> Subject: Re: [DRBD-user] Moving TeraBytes to drbd device
>
> Will you be using pacemaker for auto-recovery, or are you planning to do
> fully manual recovery?
>
> digimer
>
> On 2017-10-05 12:19 PM, José Andrés Matamoros Guevara wrote:
>> OK. That's a procedure I have thought about, but didn't know if there were
>> another more effective. I just have to check how much data I have to sync
>> during the window.
>>
>> Last question: what kind of test do you recommend to check the new drbd
>> system? I have configure a few systems but never this important.
>>
>> Thanks again,
>>
>> Andres.
>>
>>
>> -Original Message-
>> From: Digimer [mailto:li...@alteeve.ca]
>> Sent: Wednesday, 4 October, 2017 11:11 PM
>> To: José Andrés Matamoros Guevara <amatamo...@ie.com.sv>;
>> drbd-user@lists.linbit.com
>> Subject: Re: [DRBD-user] Moving TeraBytes to drbd device
>>
>> If it were up to me, given what I understand of your requirements (which is
>> minimal), I'd look at setting up the new DRBD system. Test it carefully and
>> when ready, do an rsync to get the bulk data over. When your windows opens,
>> re-run the rsync to copy just what's changed.
>>
>> If you're asking how to setup a DRBD based system, we'll need to know more
>> about what you want the system to do.
>>
>> cheers,
>>
>> digimer
>>
>> On 2017-10-04 11:29 PM, José Andrés Matamoros Guevara wrote:
>>> Thanks for your answer.
>>>
>>> Yes, I need to copy a SAN to a new drbd system. And yes, I'll check
>>> with LinBit for support if I don't find a safe solution for myself.
>>>
>>> Just a question: I have been looking through the LinBit documents and
>>> haven't found exactly what I was looking for. If you can point me
>>> about what document/documents or topics I have to read about, will be great.
>>>
>>> Thanks for the info.
>>>
>>> Andres.
>>>
>>> -Original Message-
>>> From: Digimer [mailto:li...@alteeve.ca]
>>> Sent: Tuesday, 3 October, 2017 6:10 PM
>>> To: José Andrés Matamoros Guevara <amatamo...@ie.com.sv>;
>>> drbd-user@lists.linbit.com
>>> Subject: Re: [DRBD-user] Moving TeraBytes to drbd device
>>>
>>> On 2017-10-03 06:57 PM, José Andrés Matamoros Guevara wrote:
>>>> I have been consulted about moving multiple Terabytes to a new
>>>> system using drbd to ensure high availability. I have been thinking
>>>> on multiple scenarios to move the data as fast as I can and to have
>>>> a minimal maintenance window to change the systems.
>>>>
>>>> Is there any how-to or recommendation about it? I know it is not
>>>> exactly a drbd consult but I supposed you have been using it for a
>>>> while and have more idea about a best practice. I have thought to
>>>> have a backup program to copy the data and then update it during the
>>>> maintenance window, but been lots of TB, the reading/writing data
>>>> time is going to be the factor to consider.
>>>>
>>>> Thanks in advance and best regards, Andres.
>>>
>>> The best answer is to engage LINBIT for commercial support if you've
>>> been tasked with a sensitive project and a minimal window to do it in.
>>>
>>> That said, I'll share what comes to mind;
>>>
>>> If you mean "move" as in "copy the data to a new system that already
>>> has DRBD configured and tested", then it is merely a question of hardware.
>>> Make sure you have a sufficiently fast new system to accept the
>>> incoming volume of data within the prescribed time frame. DRBD itself
>>> has minimal overhead, so i
Re: [DRBD-user] Moving TeraBytes to drbd device
Will you be using pacemaker for auto-recovery, or are you planning to do fully manual recovery? digimer On 2017-10-05 12:19 PM, José Andrés Matamoros Guevara wrote: > OK. That's a procedure I have thought about, but didn't know if there were > another more effective. I just have to check how much data I have to sync > during the window. > > Last question: what kind of test do you recommend to check the new drbd > system? I have configure a few systems but never this important. > > Thanks again, > > Andres. > > > -----Original Message- > From: Digimer [mailto:li...@alteeve.ca] > Sent: Wednesday, 4 October, 2017 11:11 PM > To: José Andrés Matamoros Guevara <amatamo...@ie.com.sv>; > drbd-user@lists.linbit.com > Subject: Re: [DRBD-user] Moving TeraBytes to drbd device > > If it were up to me, given what I understand of your requirements (which is > minimal), I'd look at setting up the new DRBD system. Test it carefully and > when ready, do an rsync to get the bulk data over. When your windows opens, > re-run the rsync to copy just what's changed. > > If you're asking how to setup a DRBD based system, we'll need to know more > about what you want the system to do. > > cheers, > > digimer > > On 2017-10-04 11:29 PM, José Andrés Matamoros Guevara wrote: >> Thanks for your answer. >> >> Yes, I need to copy a SAN to a new drbd system. And yes, I'll check >> with LinBit for support if I don't find a safe solution for myself. >> >> Just a question: I have been looking through the LinBit documents and >> haven't found exactly what I was looking for. If you can point me >> about what document/documents or topics I have to read about, will be great. >> >> Thanks for the info. >> >> Andres. >> >> -Original Message- >> From: Digimer [mailto:li...@alteeve.ca] >> Sent: Tuesday, 3 October, 2017 6:10 PM >> To: José Andrés Matamoros Guevara <amatamo...@ie.com.sv>; >> drbd-user@lists.linbit.com >> Subject: Re: [DRBD-user] Moving TeraBytes to drbd device >> >> On 2017-10-03 06:57 PM, José Andrés Matamoros Guevara wrote: >>> I have been consulted about moving multiple Terabytes to a new system >>> using drbd to ensure high availability. I have been thinking on >>> multiple scenarios to move the data as fast as I can and to have a >>> minimal maintenance window to change the systems. >>> >>> Is there any how-to or recommendation about it? I know it is not >>> exactly a drbd consult but I supposed you have been using it for a >>> while and have more idea about a best practice. I have thought to >>> have a backup program to copy the data and then update it during the >>> maintenance window, but been lots of TB, the reading/writing data >>> time is going to be the factor to consider. >>> >>> Thanks in advance and best regards, Andres. >> >> The best answer is to engage LINBIT for commercial support if you've >> been tasked with a sensitive project and a minimal window to do it in. >> >> That said, I'll share what comes to mind; >> >> If you mean "move" as in "copy the data to a new system that already >> has DRBD configured and tested", then it is merely a question of hardware. >> Make sure you have a sufficiently fast new system to accept the >> incoming volume of data within the prescribed time frame. DRBD itself >> has minimal overhead, so it is really a question of the speed of the >> disks and the replication link. >> >> If you mean to convert an existing filesystem on existing hardware to >> be backed by DRBD, then you need to either grow the backing storage by >> ~32MiB per TiB of existing data, or setup a matching sized storage >> device and configure external metadata. With that extra space, you can >> setup DRBD and it will see the existing FS and data fine. You'll need >> a full resync to the new peer, of course. Also, test/practice outside >> prod thoroughly to be certain you have the steps down pat. >> >> There are docs on how to do this openly available on LINBIT's website. >> If you get stuck on certain steps, post specific questions and we'll help. >> >> -- >> Digimer >> Papers and Projects: https://alteeve.com/w/ "I am, somehow, less >> interested in the weight and convolutions of Einstein’s brain than in >> the near certainty that people of equal talent have lived and died in >> cotton fields and sweatshops." - Stephen Jay Gould >> > > > -- > Digimer > Papers and Projects: https://al
Re: [DRBD-user] Moving TeraBytes to drbd device
If it were up to me, given what I understand of your requirements (which is minimal), I'd look at setting up the new DRBD system. Test it carefully and when ready, do an rsync to get the bulk data over. When your windows opens, re-run the rsync to copy just what's changed. If you're asking how to setup a DRBD based system, we'll need to know more about what you want the system to do. cheers, digimer On 2017-10-04 11:29 PM, José Andrés Matamoros Guevara wrote: > Thanks for your answer. > > Yes, I need to copy a SAN to a new drbd system. And yes, I'll check with > LinBit for support if I don't find a safe solution for myself. > > Just a question: I have been looking through the LinBit documents and > haven't found exactly what I was looking for. If you can point me about what > document/documents or topics I have to read about, will be great. > > Thanks for the info. > > Andres. > > -Original Message- > From: Digimer [mailto:li...@alteeve.ca] > Sent: Tuesday, 3 October, 2017 6:10 PM > To: José Andrés Matamoros Guevara <amatamo...@ie.com.sv>; > drbd-user@lists.linbit.com > Subject: Re: [DRBD-user] Moving TeraBytes to drbd device > > On 2017-10-03 06:57 PM, José Andrés Matamoros Guevara wrote: >> I have been consulted about moving multiple Terabytes to a new system >> using drbd to ensure high availability. I have been thinking on >> multiple scenarios to move the data as fast as I can and to have a >> minimal maintenance window to change the systems. >> >> Is there any how-to or recommendation about it? I know it is not >> exactly a drbd consult but I supposed you have been using it for a >> while and have more idea about a best practice. I have thought to have >> a backup program to copy the data and then update it during the >> maintenance window, but been lots of TB, the reading/writing data time >> is going to be the factor to consider. >> >> Thanks in advance and best regards, Andres. > > The best answer is to engage LINBIT for commercial support if you've been > tasked with a sensitive project and a minimal window to do it in. > > That said, I'll share what comes to mind; > > If you mean "move" as in "copy the data to a new system that already has > DRBD configured and tested", then it is merely a question of hardware. > Make sure you have a sufficiently fast new system to accept the incoming > volume of data within the prescribed time frame. DRBD itself has minimal > overhead, so it is really a question of the speed of the disks and the > replication link. > > If you mean to convert an existing filesystem on existing hardware to be > backed by DRBD, then you need to either grow the backing storage by ~32MiB > per TiB of existing data, or setup a matching sized storage device and > configure external metadata. With that extra space, you can setup DRBD and > it will see the existing FS and data fine. You'll need a full resync to the > new peer, of course. Also, test/practice outside prod thoroughly to be > certain you have the steps down pat. > > There are docs on how to do this openly available on LINBIT's website. > If you get stuck on certain steps, post specific questions and we'll help. > > -- > Digimer > Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested > in the weight and convolutions of Einstein’s brain than in the near > certainty that people of equal talent have lived and died in cotton fields > and sweatshops." - Stephen Jay Gould > -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Moving TeraBytes to drbd device
On 2017-10-03 06:57 PM, José Andrés Matamoros Guevara wrote: > I have been consulted about moving multiple Terabytes to a new system > using drbd to ensure high availability. I have been thinking on multiple > scenarios to move the data as fast as I can and to have a minimal > maintenance window to change the systems. > > Is there any how-to or recommendation about it? I know it is not exactly > a drbd consult but I supposed you have been using it for a while and > have more idea about a best practice. I have thought to have a backup > program to copy the data and then update it during the maintenance > window, but been lots of TB, the reading/writing data time is going to > be the factor to consider. > > Thanks in advance and best regards, Andres. The best answer is to engage LINBIT for commercial support if you've been tasked with a sensitive project and a minimal window to do it in. That said, I'll share what comes to mind; If you mean "move" as in "copy the data to a new system that already has DRBD configured and tested", then it is merely a question of hardware. Make sure you have a sufficiently fast new system to accept the incoming volume of data within the prescribed time frame. DRBD itself has minimal overhead, so it is really a question of the speed of the disks and the replication link. If you mean to convert an existing filesystem on existing hardware to be backed by DRBD, then you need to either grow the backing storage by ~32MiB per TiB of existing data, or setup a matching sized storage device and configure external metadata. With that extra space, you can setup DRBD and it will see the existing FS and data fine. You'll need a full resync to the new peer, of course. Also, test/practice outside prod thoroughly to be certain you have the steps down pat. There are docs on how to do this openly available on LINBIT's website. If you get stuck on certain steps, post specific questions and we'll help. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Problem updating 8.3.16 to 8.4.10 -- actually problem while *downgrading* (no valid meta-data signature found)
On 2017-09-25 07:28 AM, Lars Ellenberg wrote:
> On Sat, Sep 23, 2017 at 11:32:42PM -0400, Digimer wrote:
>> I tried updating an 8.3.19 DRBD install (on EL6.9), and when I tried to
>
> 8.3.16 is the latest I know... typo I assume.
Yup, typo. Was late...
>> connect the updated node to the older node, I got this error:
>>
>> ==[ old node ]==
>> Sep 23 23:13:47 an-a02n02 kernel: block drbd0: Handshake successful: Agreed
>> network protocol version 97
>> Sep 23 23:13:47 an-a02n02 kernel: block drbd0: conn( WFConnection ->
>> WFReportParams )
>> Sep 23 23:13:47 an-a02n02 kernel: block drbd0: Starting asender thread (from
>> drbd0_receiver [12308])
>> Sep 23 23:13:47 an-a02n02 kernel: block drbd0: incompatible after-sb-0pri
>> settings
>> Sep 23 23:13:47 an-a02n02 kernel: block drbd0: conn( WFReportParams ->
>> Disconnecting )
>> Sep 23 23:13:47 an-a02n02 kernel: block drbd0: error receiving
>> ReportProtocol, l: 25!
>> Sep 23 23:13:47 an-a02n02 kernel: block drbd1: Handshake successful: Agreed
>> network protocol version 97
>> Sep 23 23:13:47 an-a02n02 kernel: block drbd1: conn( WFConnection ->
>> WFReportParams )
>> Sep 23 23:13:47 an-a02n02 kernel: block drbd1: Starting asender thread (from
>> drbd1_receiver [7326])
>> Sep 23 23:13:47 an-a02n02 kernel: block drbd1: incompatible after-sb-0pri
>> settings
>
> You have incompatible "after-sb-0pri" settings.
I saw that, but the docs said the config was backwards compatible and
the after-sb-0pri was left as default.
>> ==[ updated node ]==
>
>> Sep 23 23:13:47 an-a02n01 kernel: drbd r0: incompatible after-sb-0pri
>> settings
>
>>
>>
>> The config is (from 'drbdadm dump'):
>>
>>
>> # /etc/drbd.conf
>> common {
>> protocol C;
>> net {
>> allow-two-primaries;
>> after-sb-0pridiscard-zero-changes;
>
> So it claims it does not like this setting,
> respectively that the handshake expected that setting
> to be identical on both nodes, but it was not.
Isn't this the default back in 8.3 days? I don't recall ever changing it...
>>
>>
>> I downgraded the node back to 8.3.16 and tried to reconnect;
>>
>>
>> Starting DRBD resources: [
>> r0
>> no suitable meta data found :(
>> Command '/sbin/drbdmeta 0 v08 /dev/sda5 internal check-resize'
>> terminated with exit code 255
>> drbdadm check-resize r0: exited with code 255
>> d(r0) 0: Failure: (119) No valid meta-data signature found.
>
> DRBD 8.4 and 8.3 have slightly different meta data "magics".
> If you want to step back, you need to "convert" the 8.4
> back to the 8.3 magic, using the 8.4 compatible drbdmeta tool,
> because, well, unsurprisingly the 8.3 drbdmeta tool does not know
> the 8.4 magics.
>
> So if you intend to downgrade to 8.3 from 8.4,
> while you still have the 8.4 tools installed,
> do: "drbdadm down all; drbdadm apply-al all"
> Only then downgrade.
Noted for the future. In this case, I recreated the MD and did a full
resync.
--
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of
Einstein’s brain than in the near certainty that people of equal talent
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
[DRBD-user] Problem updating 8.3.16 to 8.4.10
rnel: drbd r1: Starting receiver thread
(from drbd_w_r1 [21376])
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: receiver (re)started
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: conn( Unconnected ->
WFConnection )
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: Handshake successful: Agreed
network protocol version 97
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: Feature flags enabled on
protocol level: 0x0 none.
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: conn( WFConnection ->
WFReportParams )
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: Starting ack_recv thread
(from drbd_r_r0 [21381])
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: incompatible after-sb-0pri
settings
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: conn( WFReportParams ->
Disconnecting )
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: error receiving
ReportProtocol, e: -5 l: 1!
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: ack_receiver terminated
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: Terminating drbd_a_r0
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: Handshake successful: Agreed
network protocol version 97
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: Feature flags enabled on
protocol level: 0x0 none.
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: conn( WFConnection ->
WFReportParams )
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: Starting ack_recv thread
(from drbd_r_r1 [21384])
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: incompatible after-sb-0pri
settings
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: conn( WFReportParams ->
Disconnecting )
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: error receiving
ReportProtocol, e: -5 l: 1!
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: ack_receiver terminated
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: Terminating drbd_a_r1
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: Connection closed
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: conn( Disconnecting ->
StandAlone )
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: receiver terminated
Sep 23 23:13:47 an-a02n01 kernel: drbd r0: Terminating drbd_r_r0
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: Connection closed
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: conn( Disconnecting ->
StandAlone )
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: receiver terminated
Sep 23 23:13:47 an-a02n01 kernel: drbd r1: Terminating drbd_r_r1
The config is (from 'drbdadm dump'):
# /etc/drbd.conf
common {
protocol C;
net {
allow-two-primaries;
after-sb-0pridiscard-zero-changes;
after-sb-1pridiscard-secondary;
after-sb-2pridisconnect;
}
disk {
fencing resource-and-stonith;
}
syncer {
rate 30M;
}
startup {
wfc-timeout 300;
degr-wfc-timeout 120;
outdated-wfc-timeout 120;
become-primary-on both;
}
handlers {
fence-peer /usr/lib/drbd/rhcs_fence;
}
}
# resource r0 on an-a02n02.example.com: not ignored, not stacked
resource r0 {
on an-a02n01.example.com {
device /dev/drbd0 minor 0;
disk /dev/sda5;
address ipv4 10.10.20.1:7788;
meta-diskinternal;
}
on an-a02n02.example.com {
device /dev/drbd0 minor 0;
disk /dev/sda5;
address ipv4 10.10.20.2:7788;
meta-diskinternal;
}
}
# resource r1 on an-a02n02.example.com: not ignored, not stacked
resource r1 {
on an-a02n01.example.com {
device /dev/drbd1 minor 1;
disk /dev/sda6;
address ipv4 10.10.20.1:7789;
meta-diskinternal;
}
on an-a02n02.example.com {
device /dev/drbd1 minor 1;
disk /dev/sda6;
address ipv4 10.10.20.2:7789;
meta-diskinternal;
}
}
I downgraded the node back to 8.3.16 and tried to reconnect;
Starting DRBD resources: [
r0
no suitable meta data found :(
Command '/sbin/drbdmeta 0 v08 /dev/sda5 internal check-resize'
terminated with exit code 255
drbdadm check-resize r0: exited with code 255
d(r0) 0: Failure: (119) No valid meta-data signature found.
==> Use 'drbdadm create-md res' to initialize meta-data area. <==
[r0] cmd /sbin/drbdsetup 0 disk /dev/sda5 /dev/sda5 internal
--set-defaults --create-device failed - continuing!
r1
no suitable meta data found :(
Command '/sbin/drbdmeta 1 v08 /dev/sda6 internal check-resize'
terminated with exit code 255
drbdadm check-resize r1: exited with code 255
d(r1) 1: Failure: (119) No valid meta-data signature found.
==> Use 'drbdadm create-md res' to initialize meta-data area. <==
[r1] cmd /sbin/drbdsetup 1 disk /dev/sda6 /dev/sda6 internal
--set-defaults --create-device failed - continuing!
n(r0) n(r1) ].
When I try to create the MD again, it warns be that it would truncate
and cause errors, so I assume drbd 8.4 updated the metadata and 8.3
doesn't recognize
Re: [DRBD-user] Changes on drbdmanage
On 2017-09-08 10:34 AM, Julien Escario wrote: > Le 07/09/2017 à 18:43, Roland Kammerer a écrit : >> On Thu, Sep 07, 2017 at 01:48:02PM +0300, Tsirkas Georgios wrote: >>> Hello, >>> What are the changes on drbdmanage command; >> >> w00t? > > I was thinking almost the same thing. Even started writing a flame answer ;-) > > Can you unsubscride such lame users ? > > Julien It's possible the sender sent the email incomplete by accident. There is no evidence of malice so I think there is no need for a harsh reaction. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] csums-alg,verify-alg algorithm
On 2017-08-28 09:28 PM, 大川敬臣 wrote: > I'm planning to build two MySQL DB servers that synchronized by DRBD 9.0 > with RHEL 7.3. > I want to enable checksum-based synchronization by adding "csums-alg > " to drbd.conf. > After I read user's guide [5.11. Configuring checksum-based > synchronization], I had a simple question. > > The user's guide said : > " may be any message digest algorithm supported by the kernel > crypto API > in your system’s kernel configuration. > Normally, you should be able to choose at least from sha1, md5, and > crc32c." > > My question is , Is there recommendation algorithm for DB server? > The algorithms (sha1, md5, crs32) are king of old ones. Can I use sha256? > Is there some reason that sha256 is not used? > > Please give me any adovices. > > Thanks, This verifies that the blocks are the same on both nodes, not the encrypt data. So the real question is; How concerned are you that a) two blocks don't match and b) those differences are just perfectly different to cause a hash collision/false match? The stronger the algorithm, the more load it will place on the system. I would stick with something fast, maybe md5 at the most. This all said, I am not an expert. If someone else says I am wrong, believe them. :) -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD Dual Primary + GFS2 for redundant KVM hosts
On 2017-08-25 04:08 PM, Gionatan Danti wrote: > Il 25-08-2017 22:01 Digimer ha scritto: >> On 2017-08-25 03:37 PM, Gionatan Danti wrote: >> >> The overhead of clustered locking is likely such that your VM >> performance would not be good, I think. > > Mmm... I need to do some more testing with fio, it seems ;) > >> With raw clustered LVs backing the servers, you don't need cluster >> locking on a per-IO basis, only on LV create/change/delete. Because LVM >> is sitting on top of DRBD (in dual-primary), live-migration is no >> trouble at all and performance is good, too. > > True. > >> GFS2, being a cluster FS, will work fine if a node is lost, provided it >> is fenced succesfully. It's wouldn't be much of a cluster-FS >> otherwise. :) > > So no problem with quorum? A loss of a system in a two-node cluster > seems to wreack havok on other cluster filesystems (Gluster, for > example...) > > Thanks. Quorum is optional (an often misunderstood thing). https://www.alteeve.com/w/The_2-Node_Myth We've run without quorum for every system we've built over 5+ years, across dozens of sites and never once needed it. A proper fence setup, which is needed regardless, is fine. In our opinion, the complexity of a third quorum node is not justified for the limited benefit of quorum. Simplicity is simply too valuable in HA. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD Dual Primary + GFS2 for redundant KVM hosts
On 2017-08-25 03:59 PM, Gionatan Danti wrote: > Il 25-08-2017 21:46 Remolina, Diego J ha scritto: >> Danti, >> >> Have you considered using something other than drbd for VM disk >> storage? Glusterfs for example is really good for that. You may want >> to give it a try. You should look at enabling sharding from the get-go >> to speed up heals when a node goes down. >> >> I would not use Glusterfs for a file server as it's performance is >> abysmal dealing with lots of small files. I would definitively use >> DRBD for a file server as it is great in handling lots of small files >> whereas Glusterfs is horrible. But for VMs, I think gluster offers >> many niceties, easy to setup, good performance, no need to deal with >> GFS2, etc. >> >> HTH, >> >> Diego > > Hi Diego, > sure, I have an on-going discussion with the gluster mailing list about > how to do that. > > However, having been served so well by DRBD in the last 4 years, I am > somewhat reluctant to abandon it for another, probably not so > battle-tested, technology. > > In particular, it seems that to be useful (ie: stable enought) for VM > disk storage, Gluster need a 3-way cluster and sharding enabled (which I > would like to avoid). In contrast, DRBD + GFS2 (or DRBD + LVM) can be > used in a 2-way cluster without problem. > > Thanks. To me, gluster is a cloud technology solution, where DRBD is pure resiliency/replication. So for HA-focused platforms, DRDB makes a lot more sense, and that is why we use it under the Anvil! platform. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD Dual Primary + GFS2 for redundant KVM hosts
On 2017-08-25 03:37 PM, Gionatan Danti wrote: > Il 25-08-2017 14:34 Digimer ha scritto: >> >> Our Anvil! project (https://www.alteeve.com/w/Build_an_m2_Anvil!) is >> basically this, except we put the VMs on clustered LVs and use gfs2 to >> store install media and the server XML files. >> >> I would NOT put the image files on gfs2, as the distributed locking >> overhead would hurt performance a fair bit. > > Hi, I contemplated using an active/active lvm-based configuration, but I > would really like to use files as per VM disks. > So, do you feel GFS2 would be inadequate for VM disk storage? What if a > node crashes/reboots? Will GFS2 continue to work? > > Thanks. The overhead of clustered locking is likely such that your VM performance would not be good, I think. With raw clustered LVs backing the servers, you don't need cluster locking on a per-IO basis, only on LV create/change/delete. Because LVM is sitting on top of DRBD (in dual-primary), live-migration is no trouble at all and performance is good, too. GFS2, being a cluster FS, will work fine if a node is lost, provided it is fenced succesfully. It's wouldn't be much of a cluster-FS otherwise. :) -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD Dual Primary + GFS2 for redundant KVM hosts
On 2017-08-25 02:49 AM, Gionatan Danti wrote: > Hi list, > in my endless search for information :) I was testing DRBD in dual > primary mode + GFS2. The goal is "the same": running a replicated, > 2-node KVM setup where disk images are located on the GFS2 filesystem. > > I know how to integrate DRBD with corosync/pacemaker, but I wonder how > the system will perform under load. More specifically, it is my > understanding that GFS2 inevitably has some significan overhead compared > to a traditional, one-node filesystem, and this can lead to decreased > performance. > > This should be especially true when restarting (or live-migrate) a > virtual machine on the other host: as the first node has cached > significan portion of the vm disk, GFS2 will, on every first read on the > new host, fire its coherency protocol to be sure to update the first's > node in-memory cached data. The overhead should be lowered by not using > cache at all (ie: cache=none, which implies O_DIRECT), but this will > also case degraded peformance in the common case (ie: when all is > working correctly and VMs run on the first node only). > > So I ask: has some of you direct experience with a similar setup? How do > you feel about? Any other suggestions? > Thanks. Our Anvil! project (https://www.alteeve.com/w/Build_an_m2_Anvil!) is basically this, except we put the VMs on clustered LVs and use gfs2 to store install media and the server XML files. I would NOT put the image files on gfs2, as the distributed locking overhead would hurt performance a fair bit. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Dual primary and LVM
On 2017-07-26 10:14 AM, Gionatan Danti wrote: > Hi all, > I have a possibly naive question about a dual primary setup involving > LVM devices on top of DRBD. > > The main question is: using cLVM or native LVM locking, can I safely use > a LV block device on the first node, *close it*, and reopen it on the > second one? No filesystem is involved and no host is expected to > concurrently use the same LV. > > Scenario: two CentOS 7 + DRBD 8.4 nodes with LVs on top of DRBD on top > of a physical RAID array. Basically, DRBD replicate anything written to > the specific hardware array. > > Goal: having a redundant virtual machine setup, where vms can be live > migrated between the two hosts. > > Current setup: I currently run a single-primary, dual nodes setup, where > the second host has no access at all to any LV. This setup worked very > well in the past years, but it forbid using live migration (the > secondary host has no access to the LV-based vdisk attached to the vms, > so it is impossible to live migrate the running vms). > > I thought to use a dual-primary setup to have the LVs available on > *both* nodes, using a lock manager to arbitrate access to them. > > How do you see such a solution? It is workable? Or would you recommend > to use a clustered filesystem on top of the dual-primary DRBD device? > > Thanks. To clarify; The clustered locking in clvmd doesn't do anything to restrict access to the storage, it's job is only to inform other nodes that something changing on the underlying shared storage when it makes a change. To make sure a VM runs on only one node at a time, then you must use pacemaker to handle that (and stonith to prevent split-brains, but I am sure you know that already). If you use clvmd on top on DRBD, make your life simple and don't use it below DRBD at the same time. It's technically possible but it is needless complexity. You seem to already be planning this, but just to make it clear. Also note; If you don't use clvmd, then you will probably need to do a pv/vg/lvscan to pick up changes. This is quite risky though and certainly not recommended if you run DRBD in dual-primary. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] drbd-8.4.10
On 01/06/17 11:47 PM, Philipp Reisner wrote: > Hi, > > It got updated to work withthe most recent upstream kernels; a few fixes to > hard-to-trigger race conditions. No exciting news, which is good for a > "old-stable" branch. > > BTW, drbd-utils and drbd9 will follow very soon. > > > 8.4.10-1 (api:genl1/proto:86-101) > --- > * fix a race condition between attaching and writing meta-data. The impact >of the bug was destructing an object to early which leads to a crash > * fix cleanup of debugfs at rmmod time > * fix potential deadlock when trying to detach during handshake > * fix race between handshake and admin disconnect/down > * new config option "disable-write-same". That helps with backing devices >that claim to support write-same, but fail every write-same request > * compatibility with Linux kernels up to 4.12 > > > https://www.linbit.com/downloads/drbd/8.4/drbd-8.4.10-1.tar.gz > http://git.drbd.org/drbd-8.4.git/tag/drbd-8.4.10 We've updated our repo to have the 8.4.10 RPM, if anyone is interested in testing. https://www.alteeve.com/an-repo/el6/RPMS/x86_64/kmod-drbd84-8.4.10-1.el6.anvil.x86_64.rpm https://www.alteeve.com/an-repo/el6/SRPMS/drbd84-kmod-8.4.10-1.el6.anvil.src.rpm -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] drbd-8.4.10
On 01/06/17 11:47 PM, Philipp Reisner wrote: > Hi, > > It got updated to work withthe most recent upstream kernels; a few fixes to > hard-to-trigger race conditions. No exciting news, which is good for a > "old-stable" branch. > > BTW, drbd-utils and drbd9 will follow very soon. > > > 8.4.10-1 (api:genl1/proto:86-101) > --- > * fix a race condition between attaching and writing meta-data. The impact >of the bug was destructing an object to early which leads to a crash > * fix cleanup of debugfs at rmmod time > * fix potential deadlock when trying to detach during handshake > * fix race between handshake and admin disconnect/down > * new config option "disable-write-same". That helps with backing devices >that claim to support write-same, but fail every write-same request > * compatibility with Linux kernels up to 4.12 Thanks for the release! Once utils is released, how urgent would you suggest an update is? cheers -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Pacemaker DRBD dual-Primary setup, node shutdown before DRBD syncing completes.
ncing peer, I'm not > even Consistent myself.* This is so that, if the peer was alive, this node would get shot (as the peer was UpToDate). Were you sync'ing the two resources in both directions at the same time? > Apr 2 00:41:22 server7 kernel: drbd vDrbd: susp( 1 -> 0 ) > Apr 2 00:41:22 server7 kernel: block drbd0: IO ERROR: neither local nor > remote data, sector 0+0 > Apr 2 00:41:22 server7 kernel: block drbd0: IO ERROR: neither local nor > remote data, sector 344936+8 > Apr 2 00:41:22 server7 kernel: GFS2: fsid=vCluster:vGFS2.1: Error -5 > writing to log > Apr 2 00:41:22 server7 kernel: block drbd0: IO ERROR: neither local nor > remote data, sector 344944+24 > Apr 2 00:41:22 server7 kernel: GFS2: fsid=vCluster:vGFS2.1: Error -5 > writing to log > Apr 2 00:41:22 server7 kernel: block drbd0: IO ERROR: neither local nor > remote data, sector 0+0 > Apr 2 00:41:22 server7 kernel: block drbd0: IO ERROR: neither local nor > remote data, sector 344968+8 > Apr 2 00:41:22 server7 kernel: GFS2: fsid=vCluster:vGFS2.1: Error -5 > writing to log > Apr 2 00:41:22 server7 kernel: Buffer I/O error on dev dm-0, logical > block 66218, lost async page write > Apr 2 00:41:22 server7 kernel: GFS2: fsid=vCluster:vGFS2.1: Error -5 > writing to log > Apr 2 00:41:22 server7 kernel: GFS2: fsid=vCluster:vGFS2.1: Error -5 > writing to log I'm guessing this is reacting to the loss of it's backing storage. > DRBD state on surviving node server7 > --- > version: 8.4.9-1 (api:1/proto:86-101) > GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by > akemi@Build64R7, 2016-12-04 01:08:48 > 0: cs:WFConnection ro:Primary/Unknown ds:Inconsistent/DUnknown C r- > ns:3414 nr:1438774 dw:1441849 dr:72701144 al:25 bm:0 lo:0 pe:0 ua:0 > ap:0 ep:1 wo:f oos:29623116 > > > Question: > -- > Are these serious in nature? > When crashed node comes UP again and joins cluster will it cause any > problem? > How this can be avoided if a node crashes before sync completes? Obviously, I (not anyone here) can say for sure what will happen or what state your systems are in, as we don't have a complete understanding of your setup. Second, I am not an expert on DRBD or GFS2, though I use both a fair bit. My educated guess is that you're OK. Start the other node and it should reconnect and go back to sync'ing as it was before. As for how to avoid this before sync'ing is completed, well, that depends on what caused the peer to lose connection. Until the local storage is UpToDate, you're in a degraded state and can't run if the UpToDate node goes away. > Env: > - > CentOS 7.3 > DRBD 8.4 > gfs2-utils-3.1.9-3.el7.x86_64 > Pacemaker 1.1.15-11.el7_3.4 > > > Pacemaker: > - > [root@server7 ~]# pcs status > Cluster name: vCluster > Stack: corosync > Current DC: server7ha (version 1.1.15-11.el7_3.4-e174ec8) - partition > with quorum > Last updated: Sun Apr 2 01:01:43 2017 Last change: Sun Apr 2 > 00:28:39 2017 by root via cibadmin on server4ha > > 2 nodes and 9 resources configured > > Online: [ server7ha ] > OFFLINE: [ server4ha ] > > Full list of resources: > > vCluster-VirtualIP-10.168.10.199 (ocf::heartbeat:IPaddr2): > Started server7ha > vCluster-Stonith-server7ha (stonith:fence_ipmilan):Stopped > vCluster-Stonith-server4ha (stonith:fence_ipmilan):Started > server7ha > Clone Set: dlm-clone [dlm] > Started: [ server7ha ] > Stopped: [ server4ha ] > Clone Set: clvmd-clone [clvmd] > Started: [ server7ha ] > Stopped: [ server4ha ] > Master/Slave Set: drbd_data_clone [drbd_data] > Masters: [ server7ha ] > Stopped: [ server4ha ] > > Daemon Status: > corosync: active/disabled > pacemaker: active/disabled > pcsd: active/enabled > [root@server7 ~]# > > > Attaching DRBD config files. > > > --Raman -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] GFS2 - DualPrimaryDRBD hangs if a node Crashes
. So > after server7 is down and till it is completely UP the VM in server4 hangs. > > > Can anyone help how to avoid running node hang when other node crashes? > > > Attaching DRBD config file. > > > --Raman > > > > ___ > drbd-user mailing list > drbd-user@lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user > -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] drbd-9.0.7rc1 & drbd-utils-8.9.11rc1
On 23/03/17 10:20 AM, Philipp Reisner wrote: > Hi, > > Avoiding data divergence is the motto of this drbd-9 release. One the > one hand the fencing mechanism as known from the drbd-8.x world finally > works in drbd-9 as expected. Including the crm-fence-peer mechanism! > > But those who care about fencing know that it is not trivial to implement. > It is about making sure the cluster does the right thing, if the main > replication network is interrupted. That involves having a second network > for pacemaker communication or to control fencing devices. -- Quite > frequently we see that such a "second network" is just an other VLAN on > the same physical infrastructure -> Which makes the whole fencing effort > point less. > > An other approach to avoid data divergence in partitioned clusters is > to require quorum to modify the data. This approach does not need a > second network or fencing devices. But it only works if you have at > least 3 replicas. > > The most outstanding convenience feature implemented in this cycle: > intentional diskless nodes are no longer red in the output of the > status command. > > Please help testing this release candidate. > > drbd > 9.0.7rc1-1 (api:genl2/proto:86-112/transport:14) > > * various fixes to the 2-phase-commit online resize > * fix fencing and disk state transition and from Consistent; Now it is >as complete as it was under drbd-8.4; Necessary for crm-fence-peer > * fix moving previous current-uuid into bitmap slot for peer with >inconsistent disk state. Old code produced false "split-brain detected" > * fix calculation of authoritative nodes. Old code could lead to to >false "split-bain detected" > * udev workaround: Do not fail down if there is a read-only opener, instead >wait up to a second if the read-only-opener goes away. > * a primary notes if a "far away" nodes outdates itself, and records the >fact in its meta-data. > * report back to userspace if a node (or peer) is diskless by intention > * restore "Outdated" peer disk state from the meta-data upon attach > * Quorum to avoid data divergence, an alternative to fencing when the >resource has 3 or more replicas > * improve error reporting in case a promote/down/disconnect/detach >operation failes. E.g. tells which peer declined a state transition > * compiles with Linux 4.10 > > http://www.drbd.org/download/drbd/9.0/drbd-9.0.7rc1-1.tar.gz > http://git.drbd.org/drbd-9.0.git/tag/refs/tags/drbd-9.0.7rc1 > > drbd-utils > 8.9.11rc1 > > * better integration of drbd-proxy in drbd9 scenarios > * OCF resource agent and crm-fence-peer for drbd9 > * introduce quorum support > * expose information if node is intentionally diskless > * do not set peer-device-options twice > * various JSON output fixes > * udev fixes for newer ubuntu releases > * set bitmap to 0 on metadata creation > > http://www.drbd.org/download/drbd/utils/drbd-utils-8.9.11rc1.tar.gz > http://git.drbd.org/drbd-utils.git/tag/refs/tags/v8.9.11rc1 > > best, > phil YES!!! <3 -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] How long can a secondary be down?
On 14/03/17 11:32 AM, Gerald Brandt wrote: > The question is in the subject: How long can a secondary be down before > (or if) I start seeing an issue? > > Gerald There is no limit. At worst, all blocks will change and a full resync will be needed, but that doesn't cause any problems. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Concurrent local write detected!
On 02/03/17 02:40 PM, Lars Ellenberg wrote: > On Thu, Mar 02, 2017 at 03:07:52AM -0500, Digimer wrote: >> Hi all, >> >> We had an event last night on a system that's been in production for a >> couple of years; DRBD 8.3.16. At almost exactly midnight, both nodes >> threw these errors: >> >> = >> eb 28 03:42:01 aae-a01n01 rsyslogd: [origin software="rsyslogd" >> swVersion="5.8.10" x-pid="1729" x-info="http://www.rsyslog.com;] >> rsyslogd was HUPed >> Mar 2 00:00:07 aae-a01n01 kernel: block drbd0: drbd0_receiver[4763] >> Concurrent local write detected! new: 622797696s +4096; pending: >> 622797696s +4096 >> Mar 2 00:00:07 aae-a01n01 kernel: block drbd0: Concurrent write! [W >> AFTERWARDS] sec=622797696s >> Mar 2 00:00:07 aae-a01n01 kernel: block drbd0: Got DiscardAck packet >> 622797696s +4096! DRBD is not a random data generator! >> Mar 2 00:00:17 aae-a01n01 kernel: block drbd0: qemu-kvm[20305] >> Concurrent remote write detected! [DISCARD L] new: 673151680s +32768; >> pending: 673151712s +16384 > > > ... > >> [root@aae-a01n02 ~]# cat /proc/drbd >> version: 8.3.16 (api:88/proto:86-97) >> GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by >> r...@rhel6-builder-production.alteeve.ca, 2015-04-05 19:59:27 >> 0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r- >> ns:408 nr:2068182 dw:2068586 dr:48408 al:8 bm:115 lo:0 pe:0 ua:0 >> ap:0 ep:1 wo:f oos:0 >> 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r- >> ns:750365 nr:770052 dw:1520413 dr:1062911 al:15463 bm:145 lo:0 pe:0 >> ua:0 ap:0 ep:1 wo:f oos:0 > >> At this point, storage hung (I assume on purpose). Recovery was a full >> restart of the cluster. >> >> Googling doesn't return much on this. Can someone provide insight into >> what might have happened? This was a pretty scary event, and it's the >> first time I've seen it happen in all the years I've been using DRBD. >> >> Let me know if there are any other logs or info > > I guess drbd0 is your GFS2? > > What DRBD tries to tell you there is that, while one WRITE was still > "in flight", there was a new WRITE to the same LBA. > > Both nodes wrote to the exact same LBA at virtually the same time. > > GFS2 is supposed to coordinate write (and read) activity from both nodes > such that this won't happen. > > If the /proc/drbd above is from during that "storage hang", > it indicates that DRBD is not hung at all (nor any requests), > but completely "idle" (lo:0, pe:0, ua:0, ap:0 ... nothing in flight). > > If it was hung, it was not DRBD. > > In that case, my best guess is that the layer above DRBD screwed up, > and both the un-coordinated writes to the same blocks, > and the hang are symptoms of the same underlying problem. > > If you can, force an fsck, > and/or figure out what is located at those LBAs > (the s + ### are start sector + IO size in bytes). It's looking like, somehow, one of the servers was booted on both nodes at the same time... We're still trying to get access to take a closer look to confirm though. If that is what happened, it would match what you're saying (and is a massive problem). So when this happened, DRBD hung on purpose? If so, that's amazing as the hung storage appears to have saved us. madi -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
[DRBD-user] Concurrent local write detected!
: a798fa7e274428a357657fb52f0ecf40192c1985 build by r...@rhel6-builder-production.alteeve.ca, 2015-04-05 19:59:27 0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r- ns:408 nr:2068182 dw:2068586 dr:48408 al:8 bm:115 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r- ns:750365 nr:770052 dw:1520413 dr:1062911 al:15463 bm:145 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 = At this point, storage hung (I assume on purpose). Recovery was a full restart of the cluster. Googling doesn't return much on this. Can someone provide insight into what might have happened? This was a pretty scary event, and it's the first time I've seen it happen in all the years I've been using DRBD. Let me know if there are any other logs or info Thanks! digimer -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD for the upgraded kernal
What you asking for from us is not really possible for us to give. I doubt many people have used that old version in years, so asking if it will work OK with a specific kernel is nearly impossible to answer. I'll restate my comment; Build a test environment. Even a couple VMs are fine. Install the same versions of everything you have on prod, clone the images, and then try the upgrade. If you run into trouble, you can "reset" by reloading the images. There is no other way to know for sure. digimer On 20/01/17 06:14 AM, pillai bs wrote: > > Thank you Trevor/Digimer. > > > I cannot goahead and use the latest CentOS. > > Am running one application in CentOS 6.3 (My Appln will support only 6.3). > > But my requirement is to upgrade the kernal > (2.6.32-279.9.1.el6.x86_64) . Whether existing drbd > version(drbd-kmdl-2.6.32-279.el6-8.3.11-31.el6.x86_64) will support for > this Upgraded kernal or i need to upgrade my drbd version seperately? > > > Thanks, > > Madhan. > > ---- > *From:* Digimer <li...@alteeve.ca> > *Sent:* Thursday, January 19, 2017 11:08 PM > *To:* Trevor Hemsley; pillai bs; drbd-user@lists.linbit.com > *Subject:* Re: [DRBD-user] DRBD for the upgraded kernal > > On 19/01/17 08:27 AM, Trevor Hemsley wrote: >> On 19/01/17 12:55, pillai bs wrote: >>> >>> Hello Experts! >>> >>> >>> Am having cluster in below setup >>> >>> CentOS: 6.3 >>> >>> Kernal : 2.6.32-279.el6.x86_64 >>> >>> Installed drbd rpms: >>> >>> 1. drbd-kmdl-2.6.32-279.el6-8.3.11-31.el6.x86_64 >>> 2. drbd-8.3.11-31.el6.x86_64 >>> >>> The setup was in good shape. Now i have a requirement to upgrade the >>> *kernal "2.6.32-279.9.1.el6.x86_64". >>> * >>> >>> 1. Does**the above drbd version will support for this upgraded kernal >>> as well? >>> >>> if NO, from where can i get the supported drbd rpms for this upgraded >>> kernal version. >>> >>> if Yes, what are all the dependent libraries to get updated. Please >>> advise. >>> >>> >>> Thanks, >>> >>> Madhan. >>> >> >> You don't want to run either of those kernels. Both have high severity >> security vulnerabilities present and unfixed. You need to be using >> CentOS 6.8 and the latest kernel which was released last week - >> kernel-2.6.32-642.13.1.el6. You should be updating your entire system to >> 6.8 as there are numerous other high severity security vulnerabilities >> present in all versions prior to 6.8. >> >> You can find kmod-drbd84 packages in the ELRepo repository (actually >> there are drbd84 8.4.9 packages in elrepo-testing). DRBD 8.3 has been >> EOL for some years now and 8.4 is current in the 8.x series. ELRepo also >> has kmod-drbd83 (8.3.16) packages available but since 8.3 is EOL and you >> have to update anyway, it's probably better to skip and go straight to 8.4. >> >> Trevor > > To add to Trevor's comments; > > *Most* cases, you can do a clean update from 8.3 to 8.4, but a few > things did change. If you must stay on 8.3, upgrade to the latest 8.3.16 > (or 8.3.17, if it was released). > > In any case, you should always have a test cluster. If you don't, > please create one (even in VMs is better than nothing) to match the > current version of your cluster and test the upgrade there first. We > will never know your environment well enough to say "this is safe". > > If you want commercial support, LINBIT (the company behind DRBD) > offers commercial support and official releases. > > -- > Digimer > Papers and Projects: https://alteeve.com/w/ > "I am, somehow, less interested in the weight and convolutions of > Einstein’s brain than in the near certainty that people of equal talent > have lived and died in cotton fields and sweatshops." - Stephen Jay Gould > > > ___ > drbd-user mailing list > drbd-user@lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user > -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD for the upgraded kernal
On 19/01/17 08:27 AM, Trevor Hemsley wrote: > On 19/01/17 12:55, pillai bs wrote: >> >> Hello Experts! >> >> >> Am having cluster in below setup >> >> CentOS: 6.3 >> >> Kernal : 2.6.32-279.el6.x86_64 >> >> Installed drbd rpms: >> >> 1. drbd-kmdl-2.6.32-279.el6-8.3.11-31.el6.x86_64 >> 2. drbd-8.3.11-31.el6.x86_64 >> >> The setup was in good shape. Now i have a requirement to upgrade the >> *kernal "2.6.32-279.9.1.el6.x86_64". >> * >> >> 1. Does**the above drbd version will support for this upgraded kernal >> as well? >> >> if NO, from where can i get the supported drbd rpms for this upgraded >> kernal version. >> >> if Yes, what are all the dependent libraries to get updated. Please >> advise. >> >> >> Thanks, >> >> Madhan. >> > > You don't want to run either of those kernels. Both have high severity > security vulnerabilities present and unfixed. You need to be using > CentOS 6.8 and the latest kernel which was released last week - > kernel-2.6.32-642.13.1.el6. You should be updating your entire system to > 6.8 as there are numerous other high severity security vulnerabilities > present in all versions prior to 6.8. > > You can find kmod-drbd84 packages in the ELRepo repository (actually > there are drbd84 8.4.9 packages in elrepo-testing). DRBD 8.3 has been > EOL for some years now and 8.4 is current in the 8.x series. ELRepo also > has kmod-drbd83 (8.3.16) packages available but since 8.3 is EOL and you > have to update anyway, it's probably better to skip and go straight to 8.4. > > Trevor To add to Trevor's comments; *Most* cases, you can do a clean update from 8.3 to 8.4, but a few things did change. If you must stay on 8.3, upgrade to the latest 8.3.16 (or 8.3.17, if it was released). In any case, you should always have a test cluster. If you don't, please create one (even in VMs is better than nothing) to match the current version of your cluster and test the upgrade there first. We will never know your environment well enough to say "this is safe". If you want commercial support, LINBIT (the company behind DRBD) offers commercial support and official releases. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Upgarde from DRBD 8.4 to DRBD 9
On 10/01/17 04:42 AM, Samrat Priyadarshi wrote: > Hi, > > I have referred to this article. > > https://www.drbd.org/en/doc/users-guide-90/s-upgrading-drbd > > But I am using elrepo repository:- rpm -Uvh > http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm > > Can you please guide me how to upgrade to DRBD 9 using this repository. You will need to ask this on the EPEL mailing list. > If this is not possible can you give me the for this linbit repo > > [drbd-9.0] > name=DRBD 9.0 > baseurl=http://packages.linbit.com/**/yum/rhel7/drbd-9.0/ > gpgcheck=0 > > <https://www.drbd.org/en/doc/users-guide-90/s-upgrading-drbd> The hash is provided when you purchase a support agreement from LINBIT. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] secondary DRBD node shows UpToDate but is not
On 22/12/16 04:22 PM, Bart Coninckx wrote: > -Original message- > *From:* Digimer <li...@alteeve.ca> > *Sent:* Thu 22-12-2016 20:55 > *Subject:*Re: [DRBD-user] secondary DRBD node shows UpToDate but is not > *To:* Bart Coninckx <i...@bitsandtricks.com>; drbd-user@lists.linbit.com; > On 21/12/16 08:55 AM, Bart Coninckx wrote: >> Hi all, >> >> since a couple of years we have a 2 node DRBD cluster on CentOS 6.5 and >> DRBD (version: 8.4.5 (api:1/proto:86-101)) installed from source. We >> noticed that all DRBD resources show "UpToDate" on the secondary node, >> but if we restart the service, one or more resources start syncing >> again, indicating there seem to be differences in between the resources. >> This is of course not desirable. >> >> Is this a known issue with this version? If needed we will upgrade the >> cluster, but since a complete virtualization setup is running on top of >> it, this is not evident, hence the reason why I inquire first about >> possible known issues. >> >> Cheers, > > Might not be helpful, but have you tried running a verify? > > https://www.drbd.org/en/doc/users-guide-84/s-use-online-verify > > -- > Digimer > Papers and Projects: https://alteeve.ca/w/ > What if the cure for cancer is trapped in the mind of a person without > access to education? > > > > Hi, > > thank you Digimer. That seems like a decent workaround, if we automate > it. But it is just that of course. If disaster strikes in between > verifies, I believe we still have a problem, correct? > > Cheers, I've run DRBD from 8.3.7 to the latest 8.4 going back many years. Only once have I had a problem, and I am pretty sure it is because the node's storage controller acted up. DRBD should always write the same data to both nodes all the time. However, if the backing device flips some bits, then the data on the backing device might differ from what is expected. This is fundamentally a hardware problem (or low level software issue, below DRBD). Running a verify once a month is probably a good idea if you are worried, and it would be similar to how mdadm runs a verify monthly. I don't believe this is a DRBD issue itself, but DRBD can help protect against it. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] secondary DRBD node shows UpToDate but is not
On 21/12/16 08:55 AM, Bart Coninckx wrote: > Hi all, > > since a couple of years we have a 2 node DRBD cluster on CentOS 6.5 and > DRBD (version: 8.4.5 (api:1/proto:86-101)) installed from source. We > noticed that all DRBD resources show "UpToDate" on the secondary node, > but if we restart the service, one or more resources start syncing > again, indicating there seem to be differences in between the resources. > This is of course not desirable. > > Is this a known issue with this version? If needed we will upgrade the > cluster, but since a complete virtualization setup is running on top of > it, this is not evident, hence the reason why I inquire first about > possible known issues. > > Cheers, Might not be helpful, but have you tried running a verify? https://www.drbd.org/en/doc/users-guide-84/s-use-online-verify -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD packages for Proxmox 4.x
As someone who has dedicated the last five years of her life to developing open source software, let me play defence for Roland for a minute. You come up with an idea. You work your tail off to build it, make it reliable and safe, document it, and see it mature. You spend a lot of time trying to find the right balance between staying in business and keeping everything free. This is always done through some flavour of support and customization contracts. Then someone with slicker marketing or lower cost overheads comes along and uses your project to take away the customers you had hoped to cultivate. Now your wondering how you will pay your bills while someone else profits from your work without any investment of their own. How would you feel? Honestly? So you think, OK, I'll change the license to say "anyone who wants to use it, go for it! It's all yours. However, you can't as a business sell support based on my project anymore." You would think that is a pretty fair compromise... All users have no change at all - No split-level licensing or nagware. So to most everyone, zero changes. Then you have a few people start to complain. Loudly. They even start taking the debate to other forums. How long would you last before you got emotional? I very much agree that maintaining calm and civility is important. I even think that Roland's reply was emotional and not overly professional. However, he, like you and I, are human. Cut him some slack. digimer On 26/11/16 08:28 AM, Jan Schermer wrote: > What a great way to communicate with your users. > True, Linus sometimes does the same. > But you are not Linus. > So I'd advise you to think before you alienate (more) people. > This is so very unprofessional... > > Btw jasminj actually defended you if you can't see that in his first > post there[1] > and he's not wrong about [2] either. > I guess I'd better not elaborate. > Get your act together and respect your users, please. > > Jan > > [1] > https://forum.proxmox.com/threads/drbdmanage-license-change.30404/#post-152763 > [2] > https://forum.proxmox.com/threads/drbdmanage-license-change.30404/#post-152979 > > >> On 26 Nov 2016, at 09:04, Roland Kammerer <roland.kamme...@linbit.com >> <mailto:roland.kamme...@linbit.com>> wrote: >> >> On Sat, Nov 26, 2016 at 06:38:00AM +0100, Jasmin J. wrote: >>> Hello! >>> >>> There is an ongoing discussion in the Proxmox support forum >>> concerning the >>> recent license change of drbdmange: >> >> Yes, they are ongoing, no need to spread FUD here and now. Wait. >> >> But hey, as I guess this is the same JasminJ from the Proxmox forum[1], >> you can replace it in the meantime with "some perl scripts", right? Oh, >> don't forget to manage a whole cluster of nodes, support ZFS and LVM, >> and their various thin combinations and variants, provide snapshots and >> resize of resources cluster wide, and don't forget about the inner >> workings of DRBD (you already proved to be an expert in this area on >> this ML), so make sure that if you combine thin and thick in the right >> way that thin allocations don't end up thick. It's easy, just a few >> lines of perl that replace the current 67647 lines of python, right? >> >> I stop here, before I get too "friendly", but please, PLEASE, FFS don't >> come up with some BS you obviously have no clue about. >> >> Regards, rck >> >> [1] >> https://forum.proxmox.com/threads/drbdmanage-license-change.30404/#post-152979 >> ___ >> drbd-user mailing list >> drbd-user@lists.linbit.com >> http://lists.linbit.com/mailman/listinfo/drbd-user > > > > ___ > drbd-user mailing list > drbd-user@lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD9
On 28/09/16 10:53 AM, Lars Ellenberg wrote: > On Fri, Sep 23, 2016 at 05:16:32PM +0800, mzlld1988 wrote: >> Hello, everyone >> >> I have a question about using DRBD9 with pacemaker 1.1.15 . >> >> Does DRBD9 can be used in pacemaker？ > > Not in a sane way, no. > Not yet, anyways. > Various bits an pieces missing. Is there a time line for expected support? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD9
On 23/09/16 05:16 AM, mzlld1988 wrote: > Hello, everyone > > I have a question about using DRBD9 with pacemaker 1.1.15 . > > Does DRBD9 can be used in pacemaker？ Yes > Why do my cluster can't start drbd? > > Error message: > "r0 is a normal resource , and not available in stacked mode." You will need to tell us more about your setup. What OS? What are the pacemaker and DRBD configurations? Show the complete log output from both nodes (starting from just before you tried to start until a little after it failed, from both/all nodes). -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD fencing prevents resource promotion in active/passive cluster
ing-clone then start mda-ip (kind:Optional)
> (id:order-ping-clone-mda-ip-Optional)
> promote drbd1_sync then start shared_fs (kind:Mandatory)
> (id:order-drbd1_sync-shared_fs-mandatory)
> Colocation Constraints:
> ACTIVE with mda-ip (score:INFINITY) (id:colocation-ACTIVE-mda-ip-INFINITY)
> drbd1_sync with mda-ip (score:INFINITY) (rsc-role:Master)
> (with-rsc-role:Started) (id:colocation-drbd1_sync-mda-ip-INFINITY)
> shared_fs with drbd1_sync (score:INFINITY) (rsc-role:Started)
> (with-rsc-role:Master) (id:colocation-shared_fs-drbd1_sync-INFINITY)
>
> The status after starting is:
> Last updated: Fri Sep 16 14:39:57 2016 Last change: Fri Sep 16
> 14:39:49 2016 by root via cibadmin on MDA1PFP-PCS01
> Stack: corosync
> Current DC: MDA1PFP-PCS02 (version 1.1.13-10.el7-44eb2dd) - partition with
> quorum
> 2 nodes and 7 resources configured
>
> Online: [ MDA1PFP-PCS01 MDA1PFP-PCS02 ]
>
> Master/Slave Set: drbd1_sync [drbd1]
> Masters: [ MDA1PFP-PCS02 ]
> Slaves: [ MDA1PFP-PCS01 ]
> mda-ip (ocf::heartbeat:IPaddr2):Started MDA1PFP-PCS02
> Clone Set: ping-clone [ping]
> Started: [ MDA1PFP-PCS01 MDA1PFP-PCS02 ] ACTIVE (ocf::heartbeat:Dummy):
> Started MDA1PFP-PCS02
> shared_fs(ocf::heartbeat:Filesystem):Started MDA1PFP-PCS02
>
> The drbd is up as primary on MDA1PFP-PCS02 and mounted. Everything is fine.
>
> If I do any of the two tests to force a failover the resources are moved to
> the other node but the drbd is not promoted to master on the new active node:
> Last updated: Fri Sep 16 14:43:33 2016 Last change: Fri Sep 16
> 14:43:31 2016 by root via cibadmin on MDA1PFP-PCS01
> Stack: corosync
> Current DC: MDA1PFP-PCS01 (version 1.1.13-10.el7-44eb2dd) - partition with
> quorum
> 2 nodes and 7 resources configured
>
> Online: [ MDA1PFP-PCS01 ]
> OFFLINE: [ MDA1PFP-PCS02 ]
>
> Master/Slave Set: drbd1_sync [drbd1]
> Slaves: [ MDA1PFP-PCS01 ]
> mda-ip (ocf::heartbeat:IPaddr2):Started MDA1PFP-PCS01
> Clone Set: ping-clone [ping]
> Started: [ MDA1PFP-PCS01 ]
> ACTIVE (ocf::heartbeat:Dummy): Started MDA1PFP-PCS01
>
> I was able to trace this to the fencing in the drbd configuration
> MDA1PFP-S01 14:41:44 1806 0 ~ # cat /etc/drbd.d/shared_fs.res resource
> shared_fs {
> disk/dev/mapper/rhel_mdaf--pf--pep--1-drbd;
> disk {
> fencing resource-only;
> }
> handlers {
> fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
> }
> device/dev/drbd1;
> meta-disk internal;
> on MDA1PFP-S01 {
> address 192.168.123.10:7789;
> }
> on MDA1PFP-S02 {
> address 192.168.123.11:7789;
> }
> }
>
> I am using drbd 8.4.7, drbd utils 8.9.5 and pacemaker 2.3.4-7.el7 with
> corosyinc 0.9.143-15.el7 from the Centos7 repositories.
>
> MDA1PFP-S01 15:00:20 1841 0 ~ # drbdadm --version DRBDADM_BUILDTAG=GIT-hash:\
> 5d50d9fb2a967d21c0f5746370ccc066d3a67f7d\ build\ by\ mockbuild@\,\
> 2016-01-12\ 12:46:45
> DRBDADM_API_VERSION=1
> DRBD_KERNEL_VERSION_CODE=0x080407
> DRBDADM_VERSION_CODE=0x080905
> DRBDADM_VERSION=8.9.5
>
> If I disable the fencing scripts everything works as expected. If enabled, no
> node is promoted to master after failover. It seems to be a sticky
> modificaton because once a failover is simulated with fencing scripts
> activated I cannot get the cluster to work anymore. Even removing the setting
> from the DRBD configuration does not help.
>
> I discussed this on the Pacemaker mailing list and from a Pacemaker point of
> view this should not happen. The nodes are still online so no fencing should
> happen, however DRBD fences the wrong node?
>
> Best wishes,
> Jens Auer
Don't disable fencing!
You need to configure and test stonith in pacemaker. Once that's
working, then you set DRBD's fencing to 'resource-and-stonith;' and
configure the 'crm-{un,}fence-handler.sh' un/fence handlers.
With this, if a node fails (and no, redundant network links is not
enough, nodes can die in many ways), then drbd will block when the peer
is lost, call the fence handler and wait for pacemaker to report back
that the fence action was completed. This way, you will never get a
split-brain and you will get reliable recovery.
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Drbd/pacemaker active/passive san failover
On 19/09/16 03:37 AM, Marco Marino wrote:
> Hi, I'm trying to build an active/passive cluster with drbd and
> pacemaker for a san. I'm using 2 nodes with one raid controller
> (megaraid) on each one. Each node has an ssd disk that works as cache
> for read (and write?) realizing the CacheCade proprietary tecnology.
>
> Basically, the structure of the san is:
>
> Physycal disks -> RAID -> Device /dev/sdb in the OS -> Drbd resource
> (that use /dev/sdb as backend) (using pacemaker with a master/slave
> resource) -> VG (managed with pacemaker) -> Iscsi target (with
> pacemaker) -> Iscsi LUNS (one for each logical volume in the VG, managed
> with pacemaker)
>
> Few days ago, the ssd disk was wrongly removed from the primary node of
> the cluster and this caused a lot of problems: drbd resource and all
> logical volumes went in readonly mode with a lot of I/O errors but the
> cluster did not switched to the other node. All filesystem on initiators
Corosync detects node faults and causes pacemaker to trigger fencing.
Storage failures, by default, don't trigger failover. DRBD should have
marked the node with failed storage as Diskless and continued to
function be reading and writing to the good node's storage. So that's
where I would start looking for problems.
> went to readonly mode. There are 2 problems involved here (I think): 1)
> Why removing the ssd disk cause a readonly mode with I/O errors? This
Was it the FS that remounted read-only? The IO errors could have been
the OS complaining. Without logs, it's impossible to say what was
actually happening.
> means that the ssd is a single point of failure for a single node san
> with megaraid controllers and CacheCade tecnology. and 2) Why drbd
> not worked as espected?
Again, no logs means we can't say. If your array is redundant (RAID
level 1, 5 or 6) then a drive failure should be no problem. If it's not
redundant, then drbd should have been able to continue to function, as
mentioned above.
> For point 1) I'm checking with the vendor and I doubt that I can do
> something
The OS, and DRBD, will react based on what the controller's driver tells
it has happened. Why whatever happened would require pulling the logs
from the controller.
If I recall our conversation on IRC, I believe you had a supermicro box
and possibly suffered a back-plane failure. If so, then it's possible
more than just one SSD failed. Again, not something we can speak to hear
without logs and feedback from your hardware vendor's diagnostics.
> For point 2) I have errors in the drbd configuration. My idea is that
> when an I/O error happens on the primary node, the cluster should switch
> to the secondary node and shut down the damaged node.
> Here -> http://pastebin.com/79dDK66m it is possible to see the actual
> drbd configuration, but I need to change a lot of things and I want to
> share my ideas here:
You need to set 'fencing resource-and-stonith;', first of all. Secondly,
does stonith work in pacemaker? Also, setting 'degr-wfc-timeout 0;'
seems ... aggressive.
> 1) The "handlers" section should be moved in the "common" section of
> global_common.conf and not in the resource file.
Makes no real difference, but yes, it's normally in common.
> 2)I'm thinking to modify the "handlers" section as follow:
>
> handlers {
> pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;
> /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
> reboot -f";
> pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;
> /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
> reboot -f";
> local-io-error "/usr/lib/drbd/notify-io-error.sh;
> /usr/lib/drbd/notify-emergency-shutdown.sh; echo o >
> /proc/sysrq-trigger ; halt -f";
>
> # Hook into Pacemaker's fencing.
> fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
You also need to unfence-peer, iirc.
> }
>
>
> In this way, when an I/O error happens, the node will be powered off and
> pacemaker will switch resources to the other node (or at least doesn't
> create problematic behaviors...)
>
> 3) I'm thinking to move the "fencing" directive from the resource to the
> global_common.conf file. Furthermore, I want to change it to
>
> fencing resource-and-stonith;
This is what it should have been, yes. However, I doubt it would have
helped in this failure scenario.
> 4) Finally, in the global "net" section I need to add:
>
> after-sb-0pri discard-zero-changes;
> after-sb-1pri discard-secondary;
> after-sb-2pri disconnect;
>
> At the end of the work configuration will be -> http://pastebin.com/r3N1
Re: [DRBD-user] Is DRBD9 ready for production
On 11/08/16 03:10 AM, Ml Ml wrote: > Hello List, > > i wonder if DRBD9 is ready for production? > > I posted my Problem here: > http://lists.linbit.com/pipermail/drbd-user/2016-April/022893.html > > And i ran into this problem a few times now. So i switched to a 2 Node > Setup (which works fine so far). > > Everytime i go back to a 3 Node Cluster it takes only a few days and i > run into the "ASSERTION FAILED: pp_in_use" problem again. > > Please tell me if this is "just" a bug (bugs can happen) or if i > should NOT use DRBD9 for production environment. > > Thanks, > Mario The answer to this depends entirely on your tolerance for risk and the value you place on the new features. It is an X.0.y release, and one that heralded a significant rework in how DRBD works. Many people are running DRBD 9 in production just fine. So for them, it is certainly production ready. So to answer this question, I would suggest that you look at the new features in DRBD 9 versus 8.4, and see if you need those features. If so, then stick with DRBD 9 and just be extra sure to test as much as you can and plan for features not yet released. If, though, 8.4 is good enough for you and if your tolerance for risk is very low, then you might want to stick with 8.4 for the time being. Keep in mind as well; The more people that use DRBD 9, the faster it will mature. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] testing this list
On 20/07/16 12:42 PM, David Bruzos wrote: > Hi, I'm attempting to post a message to this list and it is not showing up > for some reason. Did anyone get this? > > David Test succeeded. You can pick up your celebratory cupcake by the door on the way out of the exam hall. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] drbd-8.4.8-1.tar.gz
To anyone on EL6 with interest, we've updated our DRBD RPMs on the AN!Repo[1]. digimer 1. https://alteeve.ca/an-repo/el6/ On 18/07/16 12:20 PM, Digimer wrote: > \o/ > > On 18/07/16 08:57 AM, Lars Ellenberg wrote: >> >> No one complained about the release candidate, >> so without further changes, this is the release. >> >> Git: >> http://git.linbit.com/drbd-8.4.git/shortlog/refs/tags/drbd-8.4.8-1 >> Tarball: >> http://www.drbd.org/download/drbd/8.4/drbd-8.4.8-1.tar.gz >> >>> Changes since 8.4.7: >>> >>> * compat fixes for linux kernels up to 4.7 >>> * fix potential deadlock with many parallel request on kernels >= 4.3 >>> * fix potential data divergence after write error on secondary >>>with non-synchronous replication >>> * performance improvement for kernel >= 2.6.39 >>>use request plugging again to help backends with merging batches >>> >>> Already in 8.4.7-2: >>> >>> * compat: don't trigger BUG_ON on flush in kernel < 2.6.24 (e.g. RHEL 5) >>> * al_write_transaction: skip re-scanning of bitmap page pointer array. >>>Improves performance of random writes on "huge" volumes >>>by burning less CPU and avoiding CPU cache pressure. >> >> Thanks, >> >> Lars >> > > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] drbd-8.4.8-1.tar.gz
\o/ On 18/07/16 08:57 AM, Lars Ellenberg wrote: > > No one complained about the release candidate, > so without further changes, this is the release. > > Git: > http://git.linbit.com/drbd-8.4.git/shortlog/refs/tags/drbd-8.4.8-1 > Tarball: > http://www.drbd.org/download/drbd/8.4/drbd-8.4.8-1.tar.gz > >> Changes since 8.4.7: >> >> * compat fixes for linux kernels up to 4.7 >> * fix potential deadlock with many parallel request on kernels >= 4.3 >> * fix potential data divergence after write error on secondary >>with non-synchronous replication >> * performance improvement for kernel >= 2.6.39 >>use request plugging again to help backends with merging batches >> >> Already in 8.4.7-2: >> >> * compat: don't trigger BUG_ON on flush in kernel < 2.6.24 (e.g. RHEL 5) >> * al_write_transaction: skip re-scanning of bitmap page pointer array. >>Improves performance of random writes on "huge" volumes >>by burning less CPU and avoiding CPU cache pressure. > > Thanks, > > Lars > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD 9.1 ???
On 15/07/16 12:44 AM, Igor Cicimov wrote: > On 15 Jul 2016 9:30 am, "Digimer" <li...@alteeve.ca > <mailto:li...@alteeve.ca>> wrote: >> >> On 14/07/16 07:10 PM, Igor Cicimov wrote: >> > Ok, this has been coming for a while now, does anyone know when is the >> > expected 9.1 release date? >> >> 9.0.3 just came out... So far as I know, nothing has been said about a >> release schedule. >> >> What do you mean by "this has been coming for a while now"? >> > I mean stable production ready version 9 with all the goodies announced, > like multiple active/active nodes etc., which should be version 9.1. Right?? I am sure they are working on it as best they can. It is important to remember that, unless you are a paying customer, you, I, and many others benefit greatly from LINBIT sharing their program free and open. Be patient. As an open source project, I am certain patches are always welcomed. Perhaps you can assist with some of those new features? In the meantime, the 8.4 series is still supported and extremely stable and reliable. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD 9.1 ???
On 14/07/16 07:10 PM, Igor Cicimov wrote: > Ok, this has been coming for a while now, does anyone know when is the > expected 9.1 release date? 9.0.3 just came out... So far as I know, nothing has been said about a release schedule. What do you mean by "this has been coming for a while now"? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD Recovery actions without Pacemaker
On 07/07/16 11:14 PM, Klint Gore wrote: > *From:*drbd-user-boun...@lists.linbit.com > [mailto:drbd-user-boun...@lists.linbit.com] *On Behalf Of *James Ault > *Sent:* Friday, 8 July 2016 2:02 AM > *To:* James Ault; drbd-user@lists.linbit.com > *Subject:* Re: [DRBD-user] DRBD Recovery actions without Pacemaker > > > > I see the Manual Failover section of the DRBD 8.4.x manual, and I see > that it requires that the file system be umounted before attempting to > promote and mount the file system on the secondary. > > What I meant by "those status flags" in my first message is that when a > node mounts a file system, that file system is marked as mounted > somewhere on that device. The "mounted" status flag is what I'm trying > to describe, and I'm not sure if I have the correct name for it. > > Does pacemaker or manual failover handle the case where a file server > experiences a hard failure where the umount operation is impossible? > How can the secondary copy of the file system be mounted if the umount > operation never occurred and cannot occur on server1? > > > > The basic manual failover described in the manual is when you choose > manually switch over to the other machine. You would use that if you > wanted to do maintenance on the primary. > > If the primary dies by itself, you don’t need to unmount it - that’s > where fencing comes into play. You need to make sure that the “dead” > node is well and truly dead and going to stay that way. "and going to stay that way." A bit of nuance; It's perfectly fine for the fenced node to reboot, and that is the most common configuration. All that really matters from a fence/stonith action's perspective is that the node was "off". When it boots back up (assuming it can), the software (pacemaker, drbd, etc) is in a "clean" state. It will not try to do anything related to the cluster until it rejoins. > What you’re trying to achieve is the same as what my setup is. On both > servers, nothing starts at boot and only the root file system mounts > from fstab. A script is run to make one of them primary, mount the file > systems of the drbd devices, add the external ip, start the network > services. The other node just becomes drbd secondary and that’s it. At > failover, the dead machine is pulled from the rack and taken away, then > the secondary becomes primary, mounts the file systems of the drbd > devices, adds the external ip, starts the network services. If I’m > doing manual failover for maintenance, then on the primary, the network > services are stopped, the external ip address is removed, the file > systems are unmounted, drbd is demoted to secondary. The other machine > is promoted just like hardware failover. > > Klint. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD Recovery actions without Pacemaker
On 07/07/16 07:16 AM, James Ault wrote: > > Here is a scenario: > > Two identical servers running RHEL 6.7, > Three RAID5 targets, with one Logical volume group and one logical > volume defined on top of each target. > A DRBD device defined on top of each logical volume, and then an XFS > file system defined on top of each DRBD device. > > The two identical servers are right on top of one another in the rack, > and connected by a single ethernet cable for a private network. > > The configuration works as far as synchronization between DRBD devices. > > We do NOT have pacemaker as part of this configuration at management's > request. What is the logic here? The only other viable option on EL6 is rgmanager. > We have the XFS file system mounted on server1, and this file system is > exported via NFS. > > The difficulty lies in performing failover actions without pacemaker > automation. Makes sense, given that pacemaker (and rgmanager) are resource managers. > The file system is mounted, and those status flags on the file system > are successfully mirrored to server2. What do you mean by "those status flags"? > If I disconnected all wires from server1 to simulate system failure, and > promoted server2 to primary on one of these file systems, and attempted > to mount it, the error displayed is "file system already mounted". How did you handle fencing? > I have searched the xfs_admin and mount man pages thoroughly to find an > option that would help me overcome this state. > > Our purpose of replication is to preserve and recover data in case of > failure, but we are unable to recover or use the secondary copy in our > current configuration. > > How can I recover and use this data without introducing pacemaker to our > configuration? Manually, or rgmanager. > Thanks for your help. > > -James Ault > > > _______ > drbd-user mailing list > drbd-user@lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] performance issues during a resync
Protocol A is defined like this:
Protocol A. Asynchronous replication protocol. Local write operations on
the primary node are considered completed as soon as the local disk
write has finished, and the replication packet has been placed in the
local TCP send buffer.
- https://www.drbd.org/en/doc/users-guide-84/s-replication-protocols
You probably want to look at DRBD proxy:
https://www.drbd.org/en/doc/users-guide-84/s-drbd-proxy
digimer
On 27/06/16 12:18 PM, Louis Munro wrote:
> Hello,
>
> I manage a pair of servers in two separate datacenters that replicate a
> device (backing a MySQL database) between the two.
>
> Knowing that there was a storage issue in one of the datacenters that
> meant that I/O was sometimes blocking for a few minutes, I set the
> replication protocol to A.
> That way (or so I believed), when the storage felt like acting up in our
> secondary datacenter the worse that could happen was that replication
> would lag, or even fail for a few minutes and then reconnect and catch up.
>
> Last Thursday we experienced a situation when the storage problem did
> crop up, drbd connection between the two DCs was broken and then
> replication restarted.
> The problem I faced was that during that replication, I/O latency seems
> to have been downgraded on both primary and secondary servers, just like
> I would have expected (and had previously experienced) using protocol C.
> Note that I/O was still slowed down in my secondary DC while the resync
> was taking place.
>
> It’s as though, while replication is asynchronous, resynchronizations
> are actually synchronous.
> Am I missing something?
> How can I avoid resyncs dragging down my I/O latency on the primary if
> the secondary is dragging it’s feet?
>
> Some details and logs follow.
>
>
> This is a pair of CentOS 6 servers, running drbd 8.4.5 on
> a 2.6.32-573.18.1 kernel.
>
>
> I am running with the following common configuration:
>
> common {
> protocol A;
>
> handlers {
>
> }
>
> startup {
> degr-wfc-timeout 120;
> }
>
> syncer {
> rate 500M;
> al-extents 257;
> }
> options {
> }
>
> disk {
> on-io-error detach;
> }
>
> net {
> verify-alg md5;
> }
> }
>
>
> This is what I have in the logs for the primary server:
>
> Jun 23 15:53:18 pf-db1 kernel: block drbd0: Remote failed to finish a
> request within ko-count * timeout
> Jun 23 15:53:18 pf-db1 kernel: block drbd0: peer( Secondary -> Unknown )
> conn( Connected -> Timeout ) pdsk( UpToDate -> DUnknown )
> Jun 23 15:53:18 pf-db1 kernel: drbd mysql: asender terminated
> Jun 23 15:53:18 pf-db1 kernel: drbd mysql: Terminating drbd_a_mysql
> Jun 23 15:53:18 pf-db1 kernel: block drbd0: new current UUID
> C7BC9B26D8B411FB:5386F0E7BDD5DE27:A561B9F4409AEACD:A560B9F4409AEACD
> Jun 23 15:53:18 pf-db1 kernel: drbd mysql: Connection closed
> Jun 23 15:53:18 pf-db1 kernel: drbd mysql: conn( Timeout -> Unconnected )
> Jun 23 15:53:18 pf-db1 kernel: drbd mysql: receiver terminated
> Jun 23 15:53:18 pf-db1 kernel: drbd mysql: Restarting receiver thread
> Jun 23 15:53:18 pf-db1 kernel: drbd mysql: receiver (re)started
> Jun 23 15:53:18 pf-db1 kernel: drbd mysql: conn( Unconnected ->
> WFConnection )
> Jun 23 15:53:22 pf-db1 kernel: drbd mysql: Handshake successful: Agreed
> network protocol version 101
> Jun 23 15:53:22 pf-db1 kernel: drbd mysql: Agreed to support TRIM on
> protocol level
> Jun 23 15:53:22 pf-db1 kernel: drbd mysql: conn( WFConnection ->
> WFReportParams )
> Jun 23 15:53:22 pf-db1 kernel: drbd mysql: Starting asender thread (from
> drbd_r_mysql [20164])
> Jun 23 15:53:22 pf-db1 kernel: block drbd0: drbd_sync_handshake:
> Jun 23 15:53:22 pf-db1 kernel: block drbd0: self
> C7BC9B26D8B411FB:5386F0E7BDD5DE27:A561B9F4409AEACD:A560B9F4409AEACD
> bits:1151 flags:0
> Jun 23 15:53:22 pf-db1 kernel: block drbd0: peer
> 5386F0E7BDD5DE26::A561B9F4409AEACC:A560B9F4409AEACD
> bits:0 flags:0
> Jun 23 15:53:22 pf-db1 kernel: block drbd0: uuid_compare()=1 by rule 70
> Jun 23 15:53:22 pf-db1 kernel: block drbd0: peer( Unknown -> Secondary )
> conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )
> Jun 23 15:53:22 pf-db1 kernel: block drbd0: send bitmap stats
> [Bytes(packets)]: plain 0(0), RLE 459(1), total 459; compression: 100.0%
> Jun 23 15:53:25 pf-db1 kernel: block drbd0: receive bitmap stats
> [Bytes(packets)]: plain 0(0), RLE 459(1), total 459; compression: 100.0%
> Jun 23 15:53:25 pf-db1 kernel: block drbd0: helper command:
> /sbin/drbdadm before-resync-source minor-0
> Jun 23 15:53:26 pf-db1 kernel: block drbd0: helpe
[DRBD-user] Looking for DRBD 9 RPM testers
Hi all, I noticed that there were older RPMs in ELRepo's testing[1] (EL7) for DRBD 9's kmod and utils RPMs originally built by Hiroshi Fujishima. I updated them for kmod 9.0.2-1 and utils 8.9.6. I created the drbdmanage 0.96 RPM directly from the git repo. I've not yet moved up to DRBD 9 myself, so I am not comfortable saying whether these are good or not. So this is where I was hoping some folks here might be able to help me out. Would anyone mind testing the new RPMs[2] for me? I created a repo[2] to make installing them fairly easy for anyone interested and willing to give them a test. If I can get some feedback, I'll pass them up to ELRepo (if they want them) for wider availability. Please note; These are very much testing RPMs and they are unsigned. Even if they appear to work well, I would not use them in production. :) Thanks! digimer 1. http://elrepo.org/linux/testing/el7/SRPMS/ 2. https://alteeve.ca/an-repo/testing/an-testing.repo -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Resolving split brain
On 07/06/16 04:09 PM, Lars Ellenberg wrote:
> On Tue, Jun 07, 2016 at 12:24:48PM -0400, Digimer wrote:
>> On 07/06/16 08:46 AM, David Pullman wrote:
>>> Digimer,
>>>
>>> Thanks for the direction, this sounds right for sure. So to actually do
>>> this:
>>>
>>> 1. Because we're running RHEL 6.7, I think I need to use 1.1-pcs version
>>> of the docs, chapter 8 Configure STONITH? Our nodes are supermicro based
>>> with an IPMI BMC controller, but with common non-controllable power. So
>>> I think I need to use the IPMI fencing agent?
>>
>> Yup, fence_ipmilan should work just fine. Try this, from the command line;
>>
>> fence_ipmilan -a -l -p -o status
>>
>> If you can check the state of both nodes from both nodes, then it's a
>> simple matter of adding it to pacemaker. Note that you will want to
>> configure pacemaker with 'delay="15"' for the fence method for the
>> primary node.
>>
>> This way, if comms breaks but both nodes are up, node 2 will look up how
>> to fence node 1, see the delay and sleep for 15 seconds. Node 1 looks up
>> how to fence node 2, sees no delay and shoots immediately. This way, you
>> can ensure that node 1 (assuming it's the primary node) always wins.
>>
>>> 2. Would the correct approach for the DRBD fencing and handlers be the
>>> guidance in users-guide84 under 8.3.2. Resource-level fencing using the
>>> Cluster Information Base (CIB)?
>>
>> No need to worry about the CIB directly. The pcs tool makes configuring
>> fencing in pacemaker pretty easy. Once you have fencing working in
>> pacemaker, then you can hook DRBD into it by setting 'fencing
>> resource-and-stonith' and set the fence handlers to crm-{un,}fence-peer.sh.
>>
>> With that, when DRBD loses the peer, it will block
>> (resource-and-stonith) and call the fence handler (crm-fence-peer.sh).
>> In turn, crm-fence-peer.sh asks pacemaker to shoot the lost node.
>
> Not exactly. The crm-fence-peer.sh script tries some heuristics
> on the cib content and using crmadmin, and "figures out"
> if "it is only me", or if pacemaker/"the cluster communication" also
> does not see that peer anymore.
>
> If (heuristics say that) cluster comm to that node is still up,
> we place some constraint telling pacemaker to NOT try to promote
> anyone without access to *my* data, then continue.
>
> If (heuristics say that) cluster comm to that node is also down,
>
> ... and it looks clean down (or already successfully shot),
> we place that same constraint anyways, then continue
>
> ... and you don't have pacemaker fencing enabled, there are scenarios
> where you might end up with data divergence anyways. That can only
> be avoided with fencing configured on both DRBD and pacemaker level.
>
> ... and it looks as if pacemaker will "soon" shoot that node
> (or is already in the process of doing so),
> but it has not been successfully shot yet,
> we periodically poll the cib, until that is the case
> or we hit a timeout.
>
> As of now, this script never asks pacemaker to shoot any peer.
> It may, in specific scenarios, if called with --suicide-on-failure-if-primary,
> ask pacemaker to have *this* node shot, and even tries to fall back to
> other methods of suicide.
>
> More details in said script,
> it is heavily commented and tries to be descriptive
> about not only the what, but also the why.
Oh wow, it's a lot smarter than I thought. Thanks for clarifying!
>> DRBD
>> will stay blocked until that succeeds (which is why stonith has to work
>> in pacemaker before you setup fencing in DRBD).
>
>>> Fencing. 100% required, and will prevent split brains entirely.
>
> Yes :-)
That statement I was confident in. :P
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Resolving split brain
On 03/06/16 12:38 PM, David Pullman wrote: > We have a two node cluster that has Pacemaker and DRBD 8.4 on RHEL 6 > configured in HA. The application is a monitoring system that has a web > front end and uses Postgresql as a backend. The Postgres db uses the > DRBD device for it's datastore. > > I'm trying to get a configuration that will deterministically resolve > split brain when it happens. The first priority is to have one and only > one node running the HA stack and have that updating the database with > monitoring data. The second priority is, in the event of a split brain, > to resolve to the most recent content. A *much* better approach is to build your system to not split-brain in the first place. Configure fencing/stonith in your resource manager (pacemaker or cman), then configure DRBD to hook into it via the fence-handler scripts (and set 'fencing resource-and-stonoith;'). > I've looked at the automatic split brain recovery in the docs, and tried > a couple of things but I think I'm missing something to get the > resolution in the case of two standalones. Also, I'm thinking based on > some other list entries that fencing is the way to go, but I'm not sure > how to proceed. There is no way to determine what node has "better" data. Consider; Critical but small data is written to node 1, say accounting data or something. Few inodes change but the value of that data is great. Later, on the other node, someone uploads a distro ISO. Many inodes change but they are easily replaced and have no real value. Do you discard the node with the older changes? Do you discard the node with the fewest changes? Both would result in important data being lost. > I'm getting split brain on occasion when the I/O between the nodes goes > down. We have two switches in a redundant configuration connecting the > nodes. For unrelated reasons I can't change the interconnect. > > Any suggestions, referrals to docs, etc., would be greatly appreciated. Fencing. 100% required, and will prevent split brains entirely. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD server not become as primary and sync data from other one after reboot.
On 07/05/16 06:44 AM, Pratip Ghosh wrote: > Hello Guys, > > I am a new user in this group and need some urgent help. > > I am trying to configure HA storage using DRBD with OCFS2 on Ubuntu > 14.04 at AWS EC2 instance. I configured both server as primary. If I > shutdown one server and start again it will automatically become primary > after booting process and have up-to date data in both servers, but if I > reboot any one server it was not connecting again as a primary after > booting process and showing following status in both servers. I have > restarted DRBD service in both server one by one but having same issue. > Can anyone guide me one this? > > Server1 > = > root@drbd01:~# service drbd status > drbd driver loaded OK; device status: > version: 8.4.3 (api:1/proto:86-101) > srcversion: 6551AD2C98F533733BE558C > m:res cs ro ds p mounted fstype > 0:r0 StandAlone Primary/Unknown UpToDate/DUnknown r- > > > Server2 > = > root@drbd02:~# service drbd status > drbd driver loaded OK; device status: > version: 8.4.3 (api:1/proto:86-101) > srcversion: 6551AD2C98F533733BE558C > m:res cs ro ds p mounted fstype > 0:r0 StandAlone Primary/Unknown UpToDate/DUnknown r- > > > Regards, > Pratip Ghosh This is the definition of a split-brain and it is the result of not having fencing configured. For a more useful answer, please share more information on your environment and config. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] How to detach the DRBD Block Device.
On 21/04/16 10:15 PM, Takeshi Yahagi wrote: > Hi. > > This is Yahagi(TGI). > > I use DRBD v8.3.15, now. > > I want to detach the DRBD lower device. > > But, command > "drbdadm disconnect >drbdadm detach " > is not success. > > Error Message from the command "drbdadm detach" is below > 0: State change failed: (-2) Need access to UpToDate data > Command 'drbdsetup 0 detach' terminated with exit code 17 > > I want to know the completed command sequence for detach the block device. > > Regards, > > Yahagi(TGI) > Can you paste the output of 'cat /proc/drbd' from the machine you are trying to detach? If the peer is not UpToDate and Primary, then it needs the local machine to run. (Just a guess, /proc/drbd will give better indication). madi -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Standalone configuration
On 20/04/16 02:15 PM, akan tortz wrote: > Hi ALL, > > I’m new to DRBD. > The goal is to have an asynchronous replication once or twice a day > between the nodes. > > Is this possible with DRBD without DRBD proxy solution? > > From what I’ve tested I see that the secondary node can be in a > disconnected state (drbdadm disconnect ) but I don’t know how > long. > Another concern is how much changes (writes) can accommodate the primary > node while the secondary is offline? Protocol A states that the local > writes are completed once the local disk write is finished and the > replication packet is placed in the local TCP send buffer. So that means > if TCP send buffer is full then DRBD will block all write operations? > > Thanks. The peer can be disconnected as long as you want. As inodes change, they're marked dirty (oos:X in /proc/drbd). When the peer connects, it starts copying those dirty blocks over at the resync rate (dynamic on 8.4, generally <30% of available performance *at most*). At worst, all inodes eventually change and the full device needs to be resynced. The protocols determine how changes are handled in a connected state (replication, as opposed to resync). Replication always goes as fast as possible (maximum write speed less what is being consumed during a resync operation). The protocol determines when DRBD will report the write complete to the upper level application. In protocol A, that happens as soon as the data is on the outbound network buffer. Protocol C waits until the peer confirms the data made it to persistent storage. Thus, Protocol A is async replication where protocol C is sync replication. When the peer is disconnected, no replication is attempted so the protocol has no meaning. The inode(s) are marked dirty and it's done. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] DRBD configuration with HA NFS
On 16/03/16 02:56 AM, Vijay Somkuwar wrote: > Dear Team, > > To installed SAP in HA environment, I need to create DRBD with HA NFS > mode. I had follow guide from Suse but unable to mount this volume on > both node. Link the tutorial please. What do you mean, exactly, by "unable to mount this volume on both node."? If you're trying to mount the FS you're exporting over NFS, then you probably shouldn't do that. Active/Passive HA means that it runs on one node, normally, and starts on the second node automatically if the old host node fails (after being fenced). To mount the same data on both nodes, you need a cluster-aware file system like gfs2, but that introduces a lot of complexity and a performance hit from the overhead of coordinated (clustered) locking. Even if you accept that, how would you manage which node clients connect to? > Kindly refer attached DRBD configuration file and suggest correct > configuration for DRBD. Should I able to mount this volume on both node > with Read/write access and also mount this same volume third node with > Read/Write access. > > Kindly guide for same. > > Regards, > Vijay Somkuwar > Solution Architect -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] [ClusterLabs] DRBD fencing issue on failover causes resource failure
On 16/03/16 01:51 PM, Tim Walberg wrote: > Is there a way to make this work properly without STONITH? I forgot to mention > that both nodes are virtual machines (QEMU/KVM), which makes STONITH a minor > challenge. Also, since these symptoms occur even under "pcs cluster standby", > where STONITH *shouldn't* be invoked, I'm not sure if that's the entire > answer. Not sanely, no. All clusters need HA (if you don't need to coordinate services, you don't need HA). In shared storage though, it becomes extra critical. As for fencing KVM; fence_xvm or fence_virsh (the former being ideal for production, the later being easier to setup but more fragile, so only good for testing). -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] [ClusterLabs] DRBD fencing issue on failover causes resource failure
On 16/03/16 03:29 PM, Digimer wrote: > On 16/03/16 01:51 PM, Tim Walberg wrote: >> Is there a way to make this work properly without STONITH? I forgot to >> mention >> that both nodes are virtual machines (QEMU/KVM), which makes STONITH a minor >> challenge. Also, since these symptoms occur even under "pcs cluster standby", >> where STONITH *shouldn't* be invoked, I'm not sure if that's the entire >> answer. > > Not sanely, no. All clusters need HA (if you don't need to coordinate > services, you don't need HA). In shared storage though, it becomes extra > critical. s/all cluster need HA/all clusters need fencing/ > As for fencing KVM; fence_xvm or fence_virsh (the former being ideal for > production, the later being easier to setup but more fragile, so only > good for testing). > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] [ClusterLabs] DRBD fencing issue on failover causes resource failure
provider=linbit type=drbd) >Attributes: drbd_resource=drbd0 >Operations: start interval=0s timeout=240 (drbd_dev-start-interval-0s) >promote interval=0s timeout=90 (drbd_dev-promote-interval-0s) >demote interval=0s timeout=90 (drbd_dev-demote-interval-0s) >stop interval=0s timeout=100 (drbd_dev-stop-interval-0s) >monitor interval=29s role=Master > (drbd_dev-monitor-interval-29s) >monitor interval=31s role=Slave > (drbd_dev-monitor-interval-31s) > Resource: drbd_fs (class=ocf provider=heartbeat type=Filesystem) > Attributes: device=/dev/drbd0 directory=/exports/drbd0 fstype=xfs > Operations: start interval=0s timeout=60 (drbd_fs-start-interval-0s) > stop interval=0s timeout=60 (drbd_fs-stop-interval-0s) > monitor interval=20 timeout=40 (drbd_fs-monitor-interval-20) > > Stonith Devices: > Fencing Levels: > > Location Constraints: > Ordering Constraints: > start nfsVIP then start nfs-server (kind:Mandatory) > (id:order-nfsVIP-nfs-server-mandatory) > start drbd_fs then start nfs-server (kind:Mandatory) > (id:order-drbd_fs-nfs-server-mandatory) > promote drbd_master then start drbd_fs (kind:Mandatory) > (id:order-drbd_master-drbd_fs-mandatory) > Colocation Constraints: > nfs-server with nfsVIP (score:INFINITY) > (id:colocation-nfs-server-nfsVIP-INFINITY) > nfs-server with drbd_fs (score:INFINITY) > (id:colocation-nfs-server-drbd_fs-INFINITY) > drbd_fs with drbd_master (score:INFINITY) (with-rsc-role:Master) > (id:colocation-drbd_fs-drbd_master-INFINITY) > > Resources Defaults: > resource-stickiness: 100 > failure-timeout: 60 > Operations Defaults: > No defaults set > > Cluster Properties: > cluster-infrastructure: corosync > cluster-name: nfscluster > dc-version: 1.1.13-10.el7_2.2-44eb2dd > have-watchdog: false > maintenance-mode: false > stonith-enabled: false Configure *and test* stonith in pacemaker first, then DRBD will hook into it and use it properly. DRBD simply asks pacemaker to do the fence, but you currently don't have it setup. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Split brain auto recovery with DRBD9
On 01/03/16 08:56 PM, Mark Wu wrote: > Hi Digimer, > > Thanks for your reply! > > Yes, I understand fencing can prevent split-brains. But in my case it > maybe not sufficient. Let me clarify my use case. Basically, you can > get the architecture from http://ibin.co/2YoKo5VOmRYnThe app on each > client server will do I/O via block interface directly instead using a > cluster file system. Every client server need write access to all drdb > volumes to build a VG group. But the application can guarantee that no > two servers write to the same data area at the same time because it just > writes to the data belonging to itself even when spli-brain happens. So > merging data I mentioned before means just pull all newest data from > different volumes together. > > For the suggestion of fencing, I think it still can cause somehow > out-of-sync because the fencing is invoked by cluster management > software, but in data plane the I/O could be done before the fencing > happens. > > my understanding is that fencing is invoked by the cluster management > software asynchronously. First; My experience is with 8.4, not 9. So caveat emptor. DRBD has its own concept of fencing; 'fencing; resource-and-stonith;' and 'fence-handler "/path/to/some/script";'. If DRBD detects the peer fail, it blocks IO and calls the fence handler immediately. Most often, this handler passes the request up to the cluster manager and waits for a success message. So no, DRBD will not fall out of sync. As for your use case; I honestly don't understand it. It looks rather complicated. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Split brain auto recovery with DRBD9
On 01/03/16 09:43 AM, Mark Wu wrote: > Hi guys, > > According to > http://drbd.linbit.com/users-guide-9.0/s-dual-primary-mode.html, > DRBD-9.1 will support multiple primaries. I am wondering if it can > also enhance the auto reovery from split brain. It seems that currently > we don't have a good solution of recoverying from split brain without > data loss when dual primary is configured. I think it will be good to > have a policy to merge all up-to-date data. > > I am considering to build a storage pool by aggregating remote drbd > volumes into a VG and then any client can write to LV created in the VG. > This solution can remove the limitation of single host's storage > capacity when thin provisioning is enabled. > If thin-provisioning is enabled with drbdmanage, the new-added storage > servers can't help the node which runs out of storage space. > > > That's why I care about the case of multiple primaries. Thanks for any > feedback! Fencing prevents split-brains in the first place, use it. As for auto-recovery in dual-primary; Consider that DRBD is not aware of data structures. So there is no safe way to say how to merge data back. It would have to be done at the FS-layer. If you simply said "well, sync over the node with older data", or "sync over the node with fewer changes", consider this scenario; Your accounting data is updated on one node, then sometime later, an ISO image is written to the other node. I am pretty sure that the older, smaller changes are a lot more valuable. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] A question
On 22/02/16 06:00 PM, Lars Ellenberg wrote: > On Mon, Feb 22, 2016 at 12:10:12PM -0500, Digimer wrote: >> On 22/02/16 11:59 AM, Lars Ellenberg wrote: >>> On Thu, Feb 18, 2016 at 02:04:36PM -0500, Digimer wrote: >>>> This software is purely for 8.4, so I am not worried about DRBD 9 >>>> compatibility at this time. >>>> >>>> That said, I would rather not use it anyway, so if I can get the >>>> estimate resync time via another tool, I would prefer to do so. >>> >>> I don't think we have that readily available, really. >>> It is also a rough estimate anyways, >>> based on sampling the out-of-sync count, >>> guestimating a derivative, and extrapolating to zero... >>> >>> I still could put that in some connection/peer_device specific sysfs >>> or debugfs file. >>> >>>Lars >> >> If no one else has asked for this, don't worry about it. I am pretty >> sure that it is always 'hh:mm:ss', right? It never reports days? > > Code reads: > seq_printf(seq, "finish: %lu:%02lu:%02lu", > rt / 3600, (rt % 3600) / 60, rt % 60); > > That was a "Yes." ;-) Excellent, thanks for confirming. >> Assuming so, I already went ahead and wrote a simple parser that does >> ((hh*3600) + (mm*60) + sec) and that's working well enough. >> >> Having this type of info (setup for parsing by programs) in sysfs for >> drbd 9 though would be a lovely feature to have. That is where we'll be >> focusing our long term efforts. > ___ > drbd-user mailing list > drbd-user@lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] A question
On 22/02/16 11:59 AM, Lars Ellenberg wrote: > On Thu, Feb 18, 2016 at 02:04:36PM -0500, Digimer wrote: >> This software is purely for 8.4, so I am not worried about DRBD 9 >> compatibility at this time. >> >> That said, I would rather not use it anyway, so if I can get the >> estimate resync time via another tool, I would prefer to do so. > > I don't think we have that readily available, really. > It is also a rough estimate anyways, > based on sampling the out-of-sync count, > guestimating a derivative, and extrapolating to zero... > > I still could put that in some connection/peer_device specific sysfs > or debugfs file. > >Lars If no one else has asked for this, don't worry about it. I am pretty sure that it is always 'hh:mm:ss', right? It never reports days? Assuming so, I already went ahead and wrote a simple parser that does ((hh*3600) + (mm*60) + sec) and that's working well enough. Having this type of info (setup for parsing by programs) in sysfs for drbd 9 though would be a lovely feature to have. That is where we'll be focusing our long term efforts. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] A question
This software is purely for 8.4, so I am not worried about DRBD 9 compatibility at this time. That said, I would rather not use it anyway, so if I can get the estimate resync time via another tool, I would prefer to do so. digimer On 18/02/16 02:03 PM, Julien Escario wrote: > Hello, > Using /proc/drbd doesn't sounds like a good idea : with drbd9, there's > nothing but version in there. > > Perhaps should you use drbdadm and other command line tools ? > > Regards, > Julien > > Le 18/02/2016 19:59, Digimer a écrit : >> Hi all, >> >>I'm working on a program that (amoung other things) manages DRBD. Can >> you get the estimated number of seconds remaining for a resync >> operation? /proc/drbd shows "finish: 1:19:39 speed: 3,096 (3,316) K/sec" >> and I'd like to avoid having to translate that human-readable time into >> seconds if the data is already available directly somewhere. >> >> Thanks! >> > ___ > drbd-user mailing list > drbd-user@lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
[DRBD-user] A question
Hi all, I'm working on a program that (amoung other things) manages DRBD. Can you get the estimated number of seconds remaining for a resync operation? /proc/drbd shows "finish: 1:19:39 speed: 3,096 (3,316) K/sec" and I'd like to avoid having to translate that human-readable time into seconds if the data is already available directly somewhere. Thanks! -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] Forcing a device to be primary on a backup site
On 01/02/16 09:37 AM, Kalman Meth wrote: > I have the following scenario that I am trying to simulate. > > I have a volume replicated between clusters A and B, primary on cluster A. > The volume is attached to and mounted on a VM instance VM1 on cluster A. > I take a snapshot of VM1. > I now imagine that VM1 on cluster A becomes unreachable, so I want to > bring up VM1 on cluster B, and attach to it the replicated volume. > I have to make the volume primary on cluster B. > As long as the volume is primary on cluster A, I cannot make it primary > on cluster B. > > I want to force the transition of the volume on cluster B from secondary > to primary, without having to perform operations on cluster A. > > What is the proper sequence of operations / commands to accomplish this? > > Thank you, > > - Kalman > > Kalman Z. Meth, PhD. > Cloud Platforms > IBM Haifa Research Lab > Haifa University Campus, Mount Carmel, > Haifa, 31905, Israel > e-mail: m...@il.ibm.com, Tel: +972 4 829 6341, Fax: +972 4 829 6114 Fencing handles this. Node 2 loses contact with node 1, times out, asks the cluster to fence node 1. When fencing succeeds, Node 2 knows the state of node 1 (off) and can safely promote itself without causing a split-brain and recovery can proceed. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] drbd lost lvm configuration
On 18/01/16 03:37 PM, e...@bauerfunken.de wrote:
> Hello to all,
>
> I've setup a drbd cluster of 2 nodes. Both nodes use a physical
> partition type 8e (Linux LVM) and synced by drbd successfully.
> The response of drbd-overview seems to be ok, because it say
> UpToDate/UpToDate on both nodes.
>
> Now, when I create a logical volume by pvcreate, vgcreate and lvcreate
> on this drbd-device it create this volume successfully too, but when
> I check the configuration by th following way:
>
> Primary node:
>
> vgchange -a n
> drbdadm secondary
>
> Secondary node:
You need to 'drbdadm primary res' before anything can read the
/dev/drbdX device.
> vgchange -a y
>
> Now every configuration is lost and node 2 doesn't find any volumegroup.
>
> When I change primary to the old node, pvscan doesn't find any logical
> devices.
>
> Any suggestions what's going wrong, or where I've made a mistake ?
>
> My configuration:
>
> vg0drbd.res
>
> resource vg0drbd
> {
> on filer05 {
> device /dev/drbd0;
> disk /dev/cciss/c0d0p4;
> address 192.168.2.10:7788;
> meta-disk internal;
> }
> on filer06 {
> device /dev/drbd0;
> disk /dev/cciss/c0d0p4;
> address 192.168.2.11:7788;
> meta-disk internal;
> }
> }
>
> filter in lvm.conf
>
> filter = [ "a|drbd[0-9]|", "r|.*|" ]
>
>
> Best regards
>
>
>
> Andreas Weigand
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] writer order on secondary site
On 22/12/15 11:11 AM, Mia Lueng wrote: > Hi: > I'm just wondering how secondary handle the write ordering when a same > block is written twice on primary. > > Application submits these updates: X, Y, Z. > They may or may not be to the same block. > If they are to the same block, then the application, file system > or other layer already makes sure (or at least is supposed to do that), > that the first update will finish before the second is submitted. > > These updates are replicated to the peer. > > When X,Y are to the same block, how secondary site make sure that the > first update will finish before the second is summited? > > Thanks. It will always be the same so long as the peer is connected. That is, Primary writes X, change is sent to peer, write is confirmed complete. Primary writes Y (repeat). The only thing that changes is is when the primary considers the write to be complete which depends on your protocol (c == hits storage on the peer, b == received on the peer's network, a == hits the local machines network stack). -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
Re: [DRBD-user] drbd-8.4.7 & drbd-utils-8.9.5
Woohoo!! Thanks to all! On 17/12/15 04:09 AM, Philipp Reisner wrote: > Hi, > > with this release we fixed some performance regressions in 8.4 that > trigger only when DRBD is used as block storage target. (I.e. they > do not triggered when you have a file system on top of it). > > When it comes to new features a lot was done that DRBD works better > with thinly provisioned storage and SSDs (WRITE_SAME & resync). > Look for 'rs-discard-granularity' and 'discard-zeroes-if-aligned' > on the man-pages. > > drbd > 8.4.7-1 (api:genl1/proto:86-101) > --- > * introduce WRITE_SAME support > * introduce discard-capable (or thinly allocated) device friendly resync > * improve latency on certain kernel versions and usage patterns > * improve latency of disk flushes with multiple volumes > * fix recovery handling of certain multiple error scenarios, >including possible deadlocks after certain multiple failure sequences > * introduce unfence-peer handler, >to close possible races if the replication link is flaky > * avoid potential spurious full resync by > * improve discard support, reliably zero-out/discard even with > discard_zeroes_data=0 > * update wrapper defines for linux 4.3 > * fix latency regression introduced with 8.4.5 >protocol A was sometimes synchronous, C sometimes double-latency > * avoid potential deadlock during handshake > * avoid potential deadlock in disconnect during resync >with fencing resource-and-stonith > * allow IO during some bitmap bulk IO operations > * fix "endless" transfer log walk in protocol A > > http://oss.linbit.com/drbd/8.4/drbd-8.4.7-1.tar.gz > http://git.drbd.org/drbd-8.4.git/tag/refs/tags/drbd-8.4.7 > > > drbd-utils > 8.9.5 > > * add support for new disk option 'rs-discard-granularity' (module v8.4.7) > * add support for new disk option 'discard-zeroes-if-aligned' (module v8.4.7) > * add support for v9 "path" commands > * doc improvements/corrections > * improvements to drbd ocf resource agent >and pacemaker "constraint based fencing" (crm-fence-peer.sh) > * drbd-overview improvements > > http://oss.linbit.com/drbd/drbd-utils-8.9.5.tar.gz > http://git.drbd.org/drbd-utils.git/tag/refs/tags/v8.9.5 > > > cheers, > Phil > ___ > drbd-user mailing list > drbd-user@lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ drbd-user mailing list drbd-user@lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user
