Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On Friday 09 February 2024 04:41:37 pm hw wrote: > On Fri, 2024-02-09 at 11:34 -0500, Roy J. Tellason, Sr. wrote: > > On Friday 09 February 2024 06:07:16 am hw wrote: > > > What other manufacturers could we buy UPSs from? > > > > I have a Tripp-Lite sitting next to me here that replaced an APC and > > has 2-1/2 times the capabiliity. Been in service several weeks and > > so far I'm pretty happy with it... > > They seem to be extremely rare here. Where's "here"? I ordered mine from Home Depot, online. The wait until it arrived didn't seem excessive. > Are they any good, and how's the battery availability? It seems okay, and I haven't checked on the battery availability, no need at this point and if I did it'd probably change by the time I needed them anyhow. -- Member of the toughest, meanest, deadliest, most unrelenting -- and ablest -- form of life in this section of space, a critter that can be killed but can't be tamed. --Robert A. Heinlein, "The Puppet Masters" - Information is more dangerous than cannon to a society ruled by lies. --James M Dakin
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
hw wrote: > On Fri, 2024-02-09 at 06:44 -0500, Dan Ritter wrote: > > hw wrote: > > > On Thu, 2024-02-08 at 15:29 +, Andy Smith wrote: > > > > [...] > > > That sucks. I didn't know that they don't stand behind their > > > products, and it makes APC not recommendable any longer. > > > > > > What other manufacturers could we buy UPSs from? > > > > Liebert at the high end, CyberPower at the low end. > > I've never heard of Liebert, they are rather expensive. Cyberpower > seems to be cheap. > > Are they any good, and how is the battery availability? Can they even > be monitored? Liebert is very good, and -- as you said -- expensive. If you are outfitting a datacenter, they are usually on the list. Cyberpower is reasonably reliable; the batteries can be found online. They are USB connected devices readable by NUT. Some selected stats: battery.charge: 100 battery.charge.low: 10 battery.charge.warning: 20 battery.runtime: 3060 battery.runtime.low: 300 battery.type: PbAcid battery.voltage: 24.0 battery.voltage.nominal: 24 device.mfr: CPS device.model: CST135XLU device.type: ups driver.name: usbhid-ups driver.version: 2.8.0 driver.version.data: CyberPower HID 0.6 driver.version.internal: 0.47 driver.version.usb: libusb-1.0.26 (API: 0x1000109) input.voltage: 121.0 input.voltage.nominal: 120 output.voltage: 121.0 ups.beeper.status: enabled ups.load: 16 ups.mfr: CPS ups.productid: 0501 ups.realpower.nominal: 810 ups.serial: CDQHX2004035 ups.vendorid: 0764
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On Fri, 2024-02-09 at 11:34 -0500, Roy J. Tellason, Sr. wrote: > On Friday 09 February 2024 06:07:16 am hw wrote: > > What other manufacturers could we buy UPSs from? > > I have a Tripp-Lite sitting next to me here that replaced an APC and > has 2-1/2 times the capabiliity. Been in service several weeks and > so far I'm pretty happy with it... They seem to be extremely rare here. Are they any good, and how's the battery availability?
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On Fri, 2024-02-09 at 06:44 -0500, Dan Ritter wrote: > hw wrote: > > On Thu, 2024-02-08 at 15:29 +, Andy Smith wrote: > > > [...] > > That sucks. I didn't know that they don't stand behind their > > products, and it makes APC not recommendable any longer. > > > > What other manufacturers could we buy UPSs from? > > Liebert at the high end, CyberPower at the low end. I've never heard of Liebert, they are rather expensive. Cyberpower seems to be cheap. Are they any good, and how is the battery availability? Can they even be monitored?
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
>> What other manufacturers could we buy UPSs from? > I have a Tripp-Lite sitting next to me here that replaced an APC and has > 2-1/2 times the capabiliity. Been in service several weeks and so far I'm > pretty happy with it... Would they accept a warranty claim without having to run some proprietary software (diagnostic and/or OS)? Stefan
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On Friday 09 February 2024 06:07:16 am hw wrote: > What other manufacturers could we buy UPSs from? I have a Tripp-Lite sitting next to me here that replaced an APC and has 2-1/2 times the capabiliity. Been in service several weeks and so far I'm pretty happy with it... -- Member of the toughest, meanest, deadliest, most unrelenting -- and ablest -- form of life in this section of space, a critter that can be killed but can't be tamed. --Robert A. Heinlein, "The Puppet Masters" - Information is more dangerous than cannon to a society ruled by lies. --James M Dakin
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
hw wrote: > On Thu, 2024-02-08 at 15:29 +, Andy Smith wrote: > > [...] > That sucks. I didn't know that they don't stand behind their > products, and it makes APC not recommendable any longer. > > What other manufacturers could we buy UPSs from? Liebert at the high end, CyberPower at the low end. -dsr-
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On Thu, 2024-02-08 at 15:29 +, Andy Smith wrote: > [...] > Someone on the apcupsd mailing list thinks I have a faulty UPS or > battery and should get a replacement. > > APC refuses to proceed with a warranty claim because they don't > support apcupsd or nut, only their own proprietary Powerchute. They > won't proceed unless I can get Powerchute to show these events or a > failed self-test. That sucks. I didn't know that they don't stand behind their products, and it makes APC not recommendable any longer. What other manufacturers could we buy UPSs from? > [...] > Having said that, I don't need to do a warranty claim. As it was > only purchased a couple of weeks ago, consumer law allows me to > return it to the seller as faulty whether they accept that or not, > so I'll likely do that. It's just disappointing and a lot more > hassle. That seems like the best option. You can then buy from a better manufacturer which may avoid having trouble with APC later if there's a problem.
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On 2024-02-08, Charles Curley wrote: > On Thu, 8 Feb 2024 15:29:21 + > Andy Smith wrote: > >> I do not overly want to buy a Windows licence, run it >> in a VM and pass USB through to that VM just to try this. > > You could try wine. You might need the more recent crossover-office, > which is proprietary (but contributes greatly to wine). > I wonder if he could run the app on one of these virtual machines for evaluation purposes: https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On Thu, 8 Feb 2024 15:29:21 + Andy Smith wrote: > I do not overly want to buy a Windows licence, run it > in a VM and pass USB through to that VM just to try this. You could try wine. You might need the more recent crossover-office, which is proprietary (but contributes greatly to wine). -- Does anybody read signatures any more? https://charlescurley.com https://charlescurley.com/blog/
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
Hello, On Sun, Jan 28, 2024 at 06:55:04PM +, Andy Smith wrote: > So, I must admit, I am quite tempted by BX1600MI which would cost me > about £183. The equivalent spec in the Pro range is more than twice > this price. [ TL;DR: While free software like apcupsd or nut support all APC models that you can buy today, APC (Schneider Electric) the company only supports its own Windows-only Powerchute and won't do a warranty claim unless you can run that. I therefore question the device's suitability to a Linux environment. ] Just as an update, I bought the APC Back-UPS BX1600MI and while superficially it seems fine, using "apcupsd" and/or "nut" it reports a constant stream of short-lived (less than 1 second) battery detach/re-attach and powerfail/restore events. The unit itself doesn't show any audible or visual alarm but as these events are sub-second in duration I don't know if they are just too quick for that. Someone on the apcupsd mailing list thinks I have a faulty UPS or battery and should get a replacement. APC refuses to proceed with a warranty claim because they don't support apcupsd or nut, only their own proprietary Powerchute. They won't proceed unless I can get Powerchute to show these events or a failed self-test. I can't do that because I don't have any Windows machines. I do not overly want to buy a Windows licence, run it in a VM and pass USB through to that VM just to try this. While in theory if I had heeded the warnings about Back-UPS being of lesser quality I might have bought a more expensive model that wasn't faulty (or at least did not have this problem, whatever it is), I am disappointed to learn that APC will not proceed with warranty claims unless you can run some Windows software, which puts me off the entire product range. Having said that, I don't need to do a warranty claim. As it was only purchased a couple of weeks ago, consumer law allows me to return it to the seller as faulty whether they accept that or not, so I'll likely do that. It's just disappointing and a lot more hassle. Thanks, Andy
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync--delete-after)
On 1/28/24 13:55, Andy Smith wrote: Hi, Thanks, this is very useful. On Sun, Jan 28, 2024 at 06:58:08PM +0100, hw wrote: However, stay away from their cheap models as seen on this[1] picture (Back UPS). They work and you can replace the batteries yourself even though you're not supposed to. It's a minimum basic device. It may be on ok option if you're on a budget. Their batteries last about 3 years. So, I must admit, I am quite tempted by BX1600MI which would cost me about £183. The equivalent spec in the Pro range is more than twice this price. Although the battery is not strictly user-replaceable, I watched some videos on the task and it seems pretty easily doable. Something for me to think on. Thanks, Andy I'm a fan of APC, but the consumer versions. and I don't worry about batteries until they won't last the 6 or 7 seconds it takes to spin up the 20kw kohler in the back yard. My now deceased wife was on an oxy concentrator the last 15 years of her life, and a power failure of 20 minutes might have finished her, so I bought a standby just a few months before the direcho that took power down for 3 days in June 2010. Very handy since. I have an APC-1600 that been begging for a battery for a couple years, Still works fine for those few seconds. Take care, stay well, Andy. Cheers, Gene Heskett, CET -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author, 1940) If we desire respect for the law, we must first make the law respectable. - Louis D. Brandeis
Re: Data and hardware protection measures; was: rsync --delete vs rsync --delete-after
On Sun, 28 Jan 2024 19:19:55 +0100 hw wrote: Hello hw, >How do you know in advance when the battery will have failed? Even my very basic UPS (APC Backup 1400) has a light on the front labelled "Replace Battery". That, combined with a very annoying high pitch scream, are pretty good motivators to do the job. I know the Backup 1400 was mentioned in this thread as "probably avoid" (or something similar), but it's served me well thus far. Had to replace the battery pack only once. That was after ten years, not the three to five that people have been talking about. APC no longer sell that model, but battery packs are still available. Just as an FYI, the battery packs are sealed Lead-Acid. Where I live (UK), it's possible to sell lead-acid batteries to scrap merchants. Amount paid is variable and subject to massive market forces that are best described as 'volatile'. Like others have mentioned with some of the more basic APC devices, this particular model isn't designed with user replaceable batteries in mind, but it's not an overly difficult task. It can't easily (if at all) be done leaving connected devices powered up, though. -- Regards _ "Valid sig separator is {dash}{dash}{space}" / ) "The blindingly obvious is never immediately apparent" / _)rad "Is it only me that has a working delete key?" They take away our freedom in the name of liberty Suspect Device - Stiff Little Fingers pgpqAXSSxvoLF.pgp Description: OpenPGP digital signature
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
Hi, Thanks, this is very useful. On Sun, Jan 28, 2024 at 06:58:08PM +0100, hw wrote: > However, stay away from their cheap models as seen on this[1] picture > (Back UPS). They work and you can replace the batteries yourself even > though you're not supposed to. It's a minimum basic device. It may > be on ok option if you're on a budget. Their batteries last about 3 > years. So, I must admit, I am quite tempted by BX1600MI which would cost me about £183. The equivalent spec in the Pro range is more than twice this price. Although the battery is not strictly user-replaceable, I watched some videos on the task and it seems pretty easily doable. Something for me to think on. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: Data and hardware protection measures; was: rsync --delete vs rsync --delete-after
On Fri, 2024-01-26 at 15:56 +, Michael Kjörling wrote: > On 26 Jan 2024 16:11 +0100, from h...@adminart.net (hw): > > I rather spend the money on new batteries (EUR 40 last time after 5 > > years) every couple years [...] To comment myself, I think was 3 years, not 5, sorry. > > The hardware is usually extremely difficult --- and may be impossible > > --- to replace. > > And let's not forget that you can _plan_ to perform the battery > replacement for whenever that is convenient. How do you know in advance when the battery will have failed? > Which is quite the contrast to a lightning strike blowing out even > _just_ the PSU and it needing replacement before you can even use > the computer again (and you _hope_ that nothing more took a hit, > which it probably did even if the computer _seems_ to be working > fine). It would also hit the display(s), the switches and through that everything that's connected to the network, the server(s) ... That adds up to a lot of money. > [...] > It's also worth talking to your local electrician about installing an > incoming-mains overvoltage protection for lightning protection. I > won't quote prices because I had mine installed a good while ago and > also did it together with some other electrical work, but I was > surprised at how low the cost for that was, and I _know_ that it has > saved me on at least one occasion. Hm I thought it's expensive. I'll ask when I get a chance. > [...] > > You can always tell with a good hardware RAID because it > > will indicate on the trays which disk has failed and the controller > > tells you. > > Or you can label the physical disks. Whenever I replace a disk, I > print a label with the WWN of the new disk and place it so that it is > readable without removing any disks or cabling; That doesn't exactly help when the failed disk has disappeared altogether, as if it had been removed ;) But then, you can go by the numbers of the disks you can still see. And beware of SSDs; when they fail, they're usually entirely inaccessible whereas you may be still able to resuce (some) data from a spinning disk after it failed. It's probably really bad with mainbaords that use M2 storage since apparently, they seem to support only one (of the some type at least) rather than two. So you can't use those at all. What's the point of that? ZFS cache maybe?
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On Fri, 2024-01-26 at 15:17 +, Andy Smith wrote: > Hi, > > On Fri, Jan 26, 2024 at 04:11:39PM +0100, hw wrote: > > I've never had issues with any UPS due to self tests. The batteries > > need to be replaced when they are worn out. How often that is > > required depends on the UPS and the conditions it is working in, > > usually every 3--5 years. > > Out of interest what brand of UPS do you recommend for home use that > has easily-replaceable batteries every 3–5 years? For a load of > about 300W. Generally I recommend APC because they work well (which is something to be expected and shouldn't need to be pointed out), they can easily be monitored with apcupsd and, very importantly, their batteries are usually easily available so you can replace them without difficulty. However, stay away from their cheap models as seen on this[1] picture (Back UPS). They work and you can replace the batteries yourself even though you're not supposed to. It's a minimum basic device. It may be on ok option if you're on a budget. Their batteries last about 3 years. I like the better models way better, like as on that[2] picture (Back UPS pro). I bought one a bit over 10 years ago (it even came with a 120k or so warranty for when a device protected by it would get damaged) and replaced the batteries twice so far. It's been working without any issues ever since, and it'll probably work as long as new batteries remain available. So that's about 3 years battery life as well. Then it depends on a lot of things, primarily on the availability of replacement batteries, then on how much you're willing to spend --- since you can buy used ones because the only thing that goes bad is the batteries, and you can find new old stock --- how much power you need, if you want one that features a network card and if you want a 19" rack version or a standalone version. Of course, their models change over time. The 900VA smart UPS pro delivers up to 540W, IIRC, and when it's overloaded it very annoyingly beeps, but it continues to provide power. I guess it shuts down when it's overloaded and the main power fails, but I've never had that happen yet. For only 300W you go for this one: https://www.apc.com/us/en/product/BR700G/apc-backups-pro-700va-420w-tower-120v-6x-nema-515r-outlets-avr-lcd-user-replaceable-battery/ Just keep in mind that you usually end up needing a UPS with higher capacity than you planned for. So it makes sense to check what the batterie(s) cost and what the price difference between models with lower and higher capacity is. Some models take two or more batteries while the version with lower capacity may take the same battery but only one, making it overall so much cheaper that the model with more capacity that requires two (or more) batteries may get too expensive. But there may be a model with slightly more capacity that still takes only one battery and you may be glad later that you spent a little more money for more capacity. Definitely stay away from UPSs from HP. If you can reach someone from HP at all, they will charge you before they would tell you what the price of the batteries might be :( Eaton probably makes good ones, too, but they're not common here, same as another manufacturer the name of which I can't remember. So I have no experience with them. Of course, you don't want to buy one from an unknown manufacturer with no reputation, especially when it's a chinese one. The batteries are pretty generic, but for all you know, the manufacturer may have not understood that pretty high currents can flow in an UPS and probably has skimped on the wiring and/or other components to keep it cheap, and it'll set your house on fire. APC has understood that even in their basic models (at least for the wiring; I can't tell for the other components since I don't have enough knowledge about those). After having said all the above, it's pretty simple because it comes down to that, unless anything APC is difficult to come by where you life, you can't go wrong with APC. [1]: https://cdn-reichelt.de/bilder/web/xxl_ws/E910/APC_BX1400U_01.png [2]: https://oaziscomputer.hu/images/products/6934_apc-back-ups-pro-900-br900g-gr_1527776643.jpg
Re: rsync --delete vs rsync --delete-after
On Fri, 2024-01-26 at 16:27 +, Michael Kjörling wrote: > On 26 Jan 2024 16:39 +0100, from h...@adminart.net (hw): > [...] > > Having multiple generations of backups already increases the needed > > storage space by a bit more than half. That makes it already arguable > > if it's better to make (multiple generations of) backups on a single > > RAID or on N single disks. Any of the disks can fail at any time. If > > you go with N == 2, a RAID (with multiple generations of backups on > > it) can be better because when a disk fails, the RAID will very likely > > survive and the non-RAID may not. > > I'm not sure how you figure that. It's simple: when using RAID1 with 2 disks, you double the physical storage capacity needed. When using 2 independent disks, you also double the capacity. In either case, when a disk fails, the RAID has a chance to survive the failure while the single disks don't (ignoring that you may be able to recover data from a failed disk). So either you don't loose a backup or you do loose one backup. If you're lucky, you loose the outdated backup rather than the most recent one. If you made the backup on RAID, you don't loose the most recent backup. I like it better not to loose the most recent backup. That's assuming that you have storage capacity for a single backup, i. e. one backup on RAID vs. one most recent backup on one disk and on older backup on the other disk. Of course, when you have multiple generations of backups on each set of disks, things get more complicated. It also gets more complicated when the volume you're making backups of doesn't fit on a single disk ... > [...] > > Trying to make things appear easier by pointing out that failed disks > > can be replaced is not helpful. > > It's a _backup_. _By definition_, a backup is only critical once the > primary copy becomes inaccessible for some reason. Hence: I have to disagree here. The backup is always critical before you have eliminated the possibility that the data can get lost. Only when you have done that, then you don't need a backup at all.
Re: rsync --delete vs rsync --delete-after
On Fri, 2024-01-26 at 16:11 +0100, hw wrote: > I've never had issues with any UPS due to self tests. The batteries > need to be replaced when they are worn out. How often that is > required depends on the UPS and the conditions it is working in, > usually every 3--5 years. It was with some small to mid APC model, I think. We had about 1 to 2kW worth of servers on it, so it was not that small, definitely no consumer type. When I took over maintenance somebody had configured some sort of weekly or biweekly self-test, that switched over to battery, was supposed to run the battery down to 25% or similar, and then return to mains power/charging. Except once what the UPS considered 25% charge seemingly was not, and everything shut down instantly. > I rather spend the money on new batteries (EUR 40 last time after 5 > years) every couple years rather than spending thousands on replacing > the hardware when a power surge damages it which could have been > prevented by the UPS, and it's better to have the machines shut down > properly rather taking risks with potential data loss, regardless of > file systems and RAID setups in use. I think having hardware for "thousands" and having a UPS with that cheap batteries is not that common. In above company we certainly had hardware for thousands, but changing batteries cost hundreds of Euros, even with off-brand aftermarket parts. It also was complicated to order the right parts etc. > RAID isn't as complicated as you think. Hardware RAID is most > simple, > followed by btrfs, followed by mdadm. I have to disagree with that too. Some hardware RAIDs might be simple, but others are not. Tracking down the rebrandings of Adaptec, aquisitions and mergers, is a science by itself. As is finding and installing their Firmware and utilites. Are they still calles Avago, or something new again? Or all that BBU stuff: Tracking the state of battery backup units on the controller, and ordering and replacing the correct battery is also not really easy. Clearly enterprise IT type of stuff, keeping even knowledgeable people busy for hours, if you don't do it at scale and regularily. Also often Linux support is problematic. Yes, it will work, but sometimes certain utilities are not available or work as good as with Windows. On the other hand mdadm software RAID is well documented and painless. > > With hardware RAID I can instruct someone who has no idea what > they're > doing to replace a failed disk remotely. Same goes for btrfs and > mdadm, though it better be someone who isn't entirely clueless In fact this was my job for some time: Administering hardware RAID equipped servers, and instructing "remote hands" or customers to swap harddisks. It was not always easy, not always were the correct disks pulled, even though it was correctly labelled. Sometimes clueless people tried swapping by themselves, mixing stuff up. We also had one server with wrong labelling, for whatever reason. That was no fun ;) Now I won't dispute that RAID has its place in data centers and many other applications. I just doubt that it is the correct choice for many home users. > More importantly, the hassle involved in trying to recover from a > failed disk is ridiculously enormous without RAID and can get > expensive when hours of work were lost. With RAID, you don't even > notice unless you keep an eye on it, and when a disk has failed, you > simply order a replacement and plug it in. Yes, that can happen. But more often than not the scenario is like it is with most notebooks today. You send your notebook in for repair, and have to reinstall anyway. Happened to me. I backed up my Debian system, sent the device in for hardware repair, got it back with Windows 10 ;) And no, it was not the disk that was broken, but the touchpad. > > It's not like you could go to a hardware store around the corner and > get a new disk same or next day. Even if you have a store around, > they will need to order the disk, and that can, these days, take > weeks > or months or longer if it's a small store. For consumer hard disks? I just go to my favourite shop if I need a replacement, and they've got maybe 20 or 30 types of hard disk in stock, to be bought right away. Even more with SSDs. And I am in a smallish city, pop. 250.000. > That is simply wrong. RAID doesn't protect you from malware, and > nothing protects you from user error. If you have data losses from > malware and/or user error more often than from failed disks, you're > doing something majorly wrong. In my experience user error is the main source of data loss. By far. > This shows that you have no experience with RAID and is not an > argument. I've got years of experience with RAID, both in my personal use and with employers doing stuff on RAID for customers and internal services. In my experience RAID is a nice solution for data center type setups. RAID often is problematic for home users or even small offices. > Making backups
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On Fri, 26 Jan 2024, David Wright wrote: On Fri 26 Jan 2024 at 19:03:33 (+0100), Roger Price wrote: I currently have two Eaton Ellipse ECO 1600's. ... The four screws are deeply recessed and difficult to see. They have different heads: some are Torx 10, others are a star. 20/20 hindsight might suggest that you were only intended to remove the star, if by that you mean Philips/Pozidrive. What I called "star" is probably a Quadrex. Roger
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On Fri 26 Jan 2024 at 19:03:33 (+0100), Roger Price wrote: > I currently have two Eaton Ellipse ECO 1600's. I change the batteries > every 4-5 years, but this is not as easy as it should be. It is not > evident that only one of the four back panel screws needs to be > removed. I took me a while to learn this. The four screws are deeply > recessed and difficult to see. They have different heads: some are > Torx 10, others are a star. Keep trying different screwdrivers until > you feel something turning. 20/20 hindsight might suggest that you were only intended to remove the star, if by that you mean Philips/Pozidrive. Cheers, David.
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On Fri, 26 Jan 2024, Andy Smith wrote: Out of interest what brand of UPS do you recommend for home use that has easily-replaceable batteries every 3–5 years? For a load of about 300W. I currently have two Eaton Ellipse ECO 1600's. I change the batteries every 4-5 years, but this is not as easy as it should be. It is not evident that only one of the four back panel screws needs to be removed. I took me a while to learn this. The four screws are deeply recessed and difficult to see. They have different heads: some are Torx 10, others are a star. Keep trying different screwdrivers until you feel something turning. The battery compartment is too tight. I took me 4 attempts to get the batteries back in, with the cables still connected and positioned such that the rear panel can be put back. If you re-assemble and the UPS doesn't respond to pressing the ON/OFF button, then the battery leads have detached. Start all over again. Good luck! It could have been a lot easier. Roger
Re: rsync --delete vs rsync --delete-after
On 26 Jan 2024 16:39 +0100, from h...@adminart.net (hw): >> RAID is for uptime. > > It's also for saving you from the hassle involved with loosing data > when a disk fails. Which translates to more quickly fully recovering from the loss of a storage device. When used for redundancy and staying within the redundancy threshold of your setup, _done right_, RAID reduces unplanned downtime to zero in case of loss of a storage device. Depending on your setup, the replacement can either be made online or planned for, and once complete, (hopefully) no data whatsoever has been lost and everything has happened at minimal time cost. I'm assuming that _the physical act_ of replacing the storage device is similar regardless of how it is configured in software: the same number of screws, clamps or similar need removing and reinstalling; the same amount of time is required to physically move the old and the new storage device; etc. >> If a week-long outage (to get replacement hardware and restore the >> most recent backup) and a day's worth of data loss is largely >> inconsequential, as quite frankly it likely is for most home users >> save for the cost of replacement hardware, that's a very different >> scenario from if that same outage costs $$€€¥¥ and could destroy >> your livelihood; and consequently the choices made _should_ likely >> be different. > > That's assuming your time isn't worth anything and ignores whatever > the loss of data may cost you. If that isn't relevant to you, you > don't backups, either. Sure, you can tune the RPO (recovery point objective) for your backup solution as needed. If you need a RPO of five minutes after catastrophic storage failure (meaning that you lose no more than five minutes of data regardless of when failure happens), then you're going to make different choices than if a 24-48 hour RPO is fine. But I'm willing to say that _most_ home users can recover from a loss of a day's worth of changes to their data without that being a major blow to whatever they are doing; and if the user does something important, nothing prevents running an extra backup to capture those changes. (I've done that myself on occasion.) Similarly, you are going to make different choices based on your RTO (recovery time objective). RAID gives you essentially zero RTO as long as you retain sufficient redundancy, but _not_ past that. Once your storage drops below whatever level of redundancy you have, you're looking at rebuilding from backups, at which point backup frequency and backup restoration time are the minimums which will dictate your RPO and RTO respectively. So even if you have RAID, you need to know what your target RPO and RTO are, respectively, because again those values are going to be a major factor in the design of your backup regimen. >> _Mirrored backups_ makes very little sense to me. [...] > > Having multiple generations of backups already increases the needed > storage space by a bit more than half. That makes it already arguable > if it's better to make (multiple generations of) backups on a single > RAID or on N single disks. Any of the disks can fail at any time. If > you go with N == 2, a RAID (with multiple generations of backups on > it) can be better because when a disk fails, the RAID will very likely > survive and the non-RAID may not. I'm not sure how you figure that. To survive the loss of N > 0 storage devices within a set, a storage solution needs to have a raw capacity greater than the usable capacity. An illustrative case would be a three-way mirror: it can survive the loss of two out of three storage devices, but only has the usable storage capacity of a single (typically the smallest) of the devices. It's not really any different from that a commodity HDD or SSD probably won't survive the failure of one of its platters or flash chips respectively, even ignoring cascade effects of either the failure or the cause of failure. How much extra storage space you need to keep multiple generations of backups is going to depend a lot on the rate of churn of your dataset. Again as an illustrative example only, suppose you have 1 TB of data and modify 1 MB per day, and make backups daily; with two devices each capable of storing 2 TB you can have two full copies plus 1M days' worth of history on each. This is irrespective of how those storage devices themselves are furnished. > Trying to make things appear easier by pointing out that failed disks > can be replaced is not helpful. It's a _backup_. _By definition_, a backup is only critical once the primary copy becomes inaccessible for some reason. Hence: >> The only time when something >> like mirrored backups will help you is when you have only one backup >> set, the backup itself works fine, but a backup drive fails, _and_ the >> source fails before you've been able to make a new backup. For a primary copy, _of course_ the calculus is different. -- Michael Kjörling 🔗 https://michael.kjorli
Re: Data and hardware protection measures; was: rsync --delete vs rsync --delete-after
On 26 Jan 2024 16:11 +0100, from h...@adminart.net (hw): > I rather spend the money on new batteries (EUR 40 last time after 5 > years) every couple years [...] > > The hardware is usually extremely difficult --- and may be impossible > --- to replace. And let's not forget that you can _plan_ to perform the battery replacement for whenever that is convenient. Which is quite the contrast to a lightning strike blowing out even _just_ the PSU and it needing replacement before you can even use the computer again (and you _hope_ that nothing more took a hit, which it probably did even if the computer _seems_ to be working fine). >> I've had no external power outage in the last 5 or 10 years, but a UPS >> often needs at least one battery replacement during that time. > > Outages are (still) rare here, but it suffices to trigger a fuse or > the main switch when some device shorts out, or someone working on the > solar power systems some of the neighbours have, causing crazy voltage > fluctuations, or a lightning strike somewhere in the vinicity or > whatever reason for an UPS to be required. It's also worth talking to your local electrician about installing an incoming-mains overvoltage protection for lightning protection. I won't quote prices because I had mine installed a good while ago and also did it together with some other electrical work, but I was surprised at how low the cost for that was, and I _know_ that it has saved me on at least one occasion. It won't do power conditioning or power loss protection of course, but it _does_ greatly increase the odds that your home wiring survives a lightning-related voltage surge. (Nothing will realistically protect you against a _direct_ lightning strike; in that case the very best you can hope for is damage containment.) > More importantly, the hassle involved in trying to recover from a > failed disk is ridiculously enormous without RAID and can get > expensive when hours of work were lost. With RAID, you don't even > notice unless you keep an eye on it, and when a disk has failed, you > simply order a replacement and plug it in. Indeed; the point of RAID is uptime. > You can always tell with a good hardware RAID because it > will indicate on the trays which disk has failed and the controller > tells you. Or you can label the physical disks. Whenever I replace a disk, I print a label with the WWN of the new disk and place it so that it is readable without removing any disks or cabling; then I use the WWN to identify the disk in software; in both cases because the WWN is a stable identifier that I can fully expect will never change throughout the disk's lifetime. So when the system tells me that wwn-0x123456789abcdef0 is having issues, I can quickly and accurately identify the exact physical device that needs replacement once I have a replacement on hand. And if the kernel logs are telling me that, say, sdg is having issues, I can map that back to whatever WWN happens to map to that identifier at that particular time. (In practice, I'm more likely to get useful error details through ZFS status monitoring tools, where I already use the WWN, so I likely won't need to go that somewhat circuitous route.) > Yes, my setup is far from ideal when it comes to backups in that I > should make backups more frequently. That doesn't mean I shouldn't > have good backups and that UPSs and RAID were not required. Or that, again, they solve different problems. -- Michael Kjörling 🔗 https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”
Re: rsync --delete vs rsync --delete-after
On Thu, 2024-01-18 at 13:09 +, Michael Kjörling wrote: > On 18 Jan 2024 13:26 +0100, from r...@h5.or.at (Ralph Aichinger): > > As a home/SOHO user, I'd rather have a working backup every few hours > > or every day than some RAID10 wonder > > Definitely agree that a solid backup regimen (including regular > automated backups; at least one off-site copy _at least_ of critical, > hot data; and planning for the contingency that you need to restore > that backup onto a brand new system without access to anything on your > current system -- think "home burns down at night" or "burglar" > scenario) is the _first_ step, and one that a great deal of people > still fail at. > > RAID is for uptime. It's also for saving you from the hassle involved with loosing data when a disk fails. > If a week-long outage (to get replacement hardware and restore the > most recent backup) and a day's worth of data loss is largely > inconsequential, as quite frankly it likely is for most home users > save for the cost of replacement hardware, that's a very different > scenario from if that same outage costs $$€€¥¥ and could destroy > your livelihood; and consequently the choices made _should_ likely > be different. That's assuming your time isn't worth anything and ignores whatever the loss of data may cost you. If that isn't relevant to you, you don't backups, either. > _Mirrored backups_ makes very little sense to me. If a storage device > used for storage of backups fails prematurely, just toss it and get a > new one and make a new backup. If the backup software goes haywire and > starts overwriting everything with random garbage, having the garbage > mirrored isn't going to help you. It's much better to have two > independent backup targets and switching between them, and figuring > the switching interval into your RPO. The only time when something > like mirrored backups will help you is when you have only one backup > set, the backup itself works fine, but a backup drive fails, _and_ the > source fails before you've been able to make a new backup. That's a > _very_ narrow scenario and easily solved by having two backup sets. Having multiple generations of backups already increases the needed storage space by a bit more than half. That makes it already arguable if it's better to make (multiple generations of) backups on a single RAID or on N single disks. Any of the disks can fail at any time. If you go with N == 2, a RAID (with multiple generations of backups on it) can be better because when a disk fails, the RAID will very likely survive and the non-RAID may not. So for daily use, I'd ideally make multiple generations on RAID and copy the latest generation to somewhere off-site, using either single disks, or another RAID. Having no hassle and no downtime at the production system and instead having hassle and downtime off-site may be much more desirable than having it the other way round. Trying to make things appear easier by pointing out that failed disks can be replaced is not helpful. Replacing a disk in a RAID isn't any more difficult (and can be much easier) than replacing a disk that isn't in a RAID.
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
On Fri, 26 Jan 2024, Andy Smith wrote: > Hi, > > On Fri, Jan 26, 2024 at 04:11:39PM +0100, hw wrote: >> I've never had issues with any UPS due to self tests. The batteries >> need to be replaced when they are worn out. How often that is >> required depends on the UPS and the conditions it is working in, >> usually every 3--5 years. > > Out of interest what brand of UPS do you recommend for home use that > has easily-replaceable batteries every 3–5 years? For a load of > about 300W. just my experience i have 2 apc 740's that date from the mid 90's when the internal battery gave out i added a 50 amp anderson connector i connect to a 35ah sla you can change battteries while it's running
Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after)
Hi, On Fri, Jan 26, 2024 at 04:11:39PM +0100, hw wrote: > I've never had issues with any UPS due to self tests. The batteries > need to be replaced when they are worn out. How often that is > required depends on the UPS and the conditions it is working in, > usually every 3--5 years. Out of interest what brand of UPS do you recommend for home use that has easily-replaceable batteries every 3–5 years? For a load of about 300W. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: rsync --delete vs rsync --delete-after
On Thu, 2024-01-18 at 13:26 +0100, Ralph Aichinger wrote: > Hello fellow Debian users, > > On Thu, 2024-01-18 at 12:18 +0100, hw wrote: > > > Always use an UPS. > > > Here I have a somewhat contrarian view, I hope not to offend too much: It's not offending, you merely have a different opinion. > For countries with stable electricity supplies (like Austria where I > live) having a small UPS might actually lead to more problems instead > of less, unless you are putting a lot of effort into it. Very often > have I had problems with UPSes, e.g. batteries dying, the UPS going > into some self test mode and inadvertedly shutting down, etc. I've never had issues with any UPS due to self tests. The batteries need to be replaced when they are worn out. How often that is required depends on the UPS and the conditions it is working in, usually every 3--5 years. I rather spend the money on new batteries (EUR 40 last time after 5 years) every couple years rather than spending thousands on replacing the hardware when a power surge damages it which could have been prevented by the UPS, and it's better to have the machines shut down properly rather taking risks with potential data loss, regardless of file systems and RAID setups in use. The hardware is usually extremely difficult --- and may be impossible --- to replace. On top of that, noone pays for the time and effort lost, and even someone did, I'd rather keep my hardware instead of going to all the lengths required to replace it. Loosing hours of work due to power outages can also be an issue, and nobody pays for that, either. > I've had no external power outage in the last 5 or 10 years, but a UPS > often needs at least one battery replacement during that time. Outages are (still) rare here, but it suffices to trigger a fuse or the main switch when some device shorts out, or someone working on the solar power systems some of the neighbours have, causing crazy voltage fluctuations, or a lightning strike somewhere in the vinicity or whatever reason for an UPS to be required. Way too many times than I could count my UPSs saved me and the hardware from power supply problems, so I can't recommend to go without one. It doesn't matter how stable you think the power grid in your country is. > [...] > Here I also doubt if this is a wise suggestion for the typical home > or small office user. RAID leads to lots and lots of complexity, that > is often not needed in a home setup. RAID isn't as complicated as you think. Hardware RAID is most simple, followed by btrfs, followed by mdadm. With hardware RAID I can instruct someone who has no idea what they're doing to replace a failed disk remotely. Same goes for btrfs and mdadm, though it better be someone who isn't entirely clueless. More importantly, the hassle involved in trying to recover from a failed disk is ridiculously enormous without RAID and can get expensive when hours of work were lost. With RAID, you don't even notice unless you keep an eye on it, and when a disk has failed, you simply order a replacement and plug it in. It's not like you could go to a hardware store around the corner and get a new disk same or next day. Even if you have a store around, they will need to order the disk, and that can, these days, take weeks or months or longer if it's a small store. The only way to get a replacement is ordering online, which requires that your computer still works and that you have access to your passwords. > I'd rather have a working backup setup with many independent copies > before I even start thinking about RAID. Yes, disks can fail, but > data loss often is due to user error and malware. That is simply wrong. RAID doesn't protect you from malware, and nothing protects you from user error. If you have data losses from malware and/or user error more often than from failed disks, you're doing something majorly wrong. > RAID helps very little with the latter two causes of data loss. And > all too often have I seen people mess up their complicated RAID > setups, because they pulled the wrong disk when another one broke, > or because they misinterpreted complicated error messages, creating > unnecessary data loss out of user error by themselves. This shows that you have no experience with RAID and is not an argument. Making backups is way more complicated than RAID. You can way more easily overwrite the wrong backup or misinterpret error messages of your backup solution than you can pull the wrong disk from a RAID or misinterpret error messages from your RAID. How exactly would you pull the wrong disk from a RAID and thus cause data loss? Before you pull one, you make a backup. When the disk has been pulled, its contents remain unchanged and when you put it back in, your data is still there --- plus you have the backup. Sure it can sometimes be difficult to tell which disk you need to replace, and it's not an issue because you can always figure out which one you need to replace.
Re: rsync --delete vs rsync --delete-after
On 2024-01-18, Andy Smith wrote: > Could check the man page then like I said. > > Some options require rsync to know the full file list, so these > options disable the incremental recursion mode. These include: > --delete-before, --delete-after, --prune-empty-dirs, and > --delay-updates. You are right, thanks !
Re: rsync --delete vs rsync --delete-after
On Thu, 2024-01-18 at 21:44 +, Andy Smith wrote: > Hello, > > On Thu, Jan 18, 2024 at 10:01:46PM +0100, Michel Verdier wrote: > > On 2024-01-18, Andy Smith wrote: > > > If you use --delete-after (and some other options) then rsync has > > > to > > > check every file before it can do any work, whereas normally it > > > will > > > find a few files to work on and start work, meanwhile > > > incrementally > > > scanning for more. > > > > Not sure of that. > > Could check the man page then like I said. > > Some options require rsync to know the full file list, so these > options disable the incremental recursion mode. These include: > --delete-before, --delete-after, --prune-empty-dirs, and > --delay-updates. > > Thanks, > Andy > Hi, OP here. Just an update. Last night I did the usual copy from the primary backup drive to the secondary backup drive, except I did it using rsync --delete instead of rsync --delete-after. This time, it took 73 minutes. The previous time it took 131 minutes. So it did go significantly quicker. And "seemed" to go okay. Thank you to all who have weighed in on this topic!
Re: rsync --delete vs rsync --delete-after
Hello, On Thu, Jan 18, 2024 at 10:01:46PM +0100, Michel Verdier wrote: > On 2024-01-18, Andy Smith wrote: > > If you use --delete-after (and some other options) then rsync has to > > check every file before it can do any work, whereas normally it will > > find a few files to work on and start work, meanwhile incrementally > > scanning for more. > > Not sure of that. Could check the man page then like I said. Some options require rsync to know the full file list, so these options disable the incremental recursion mode. These include: --delete-before, --delete-after, --prune-empty-dirs, and --delay-updates. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: rsync --delete vs rsync --delete-after
On 2024-01-18, Andy Smith wrote: > If you use --delete-after (and some other options) then rsync has to > check every file before it can do any work, whereas normally it will > find a few files to work on and start work, meanwhile incrementally > scanning for more. Not sure of that. rsync always gets a list of files on both sides and compares them before doing anything. But --delete-after forces a second pass on the target for deletion instead of doing it during copying. So a bit more IO.
Re: rsync --delete vs rsync --delete-after
>> > However, I have read that using rsync --delete instead of rsync -- >> > delete-after is faster and uses less memory, and so is more efficient. >> I'd be surprised if it makes a significant difference. > If you use --delete-after (and some other options) then rsync has to > check every file before it can do any work, Oh, right, I was thinking of `--delete-delay`. [ Tho, as you mention, the performance impact of `--delete-after` is also negligible in many cases. ] Stefan
Re: rsync --delete vs rsync --delete-after
Hello, On Wed, Jan 17, 2024 at 11:09:35PM -0500, Stefan Monnier wrote: > > However, I have read that using rsync --delete instead of rsync -- > > delete-after is faster and uses less memory, and so is more efficient. > > I'd be surprised if it makes a significant difference. If you use --delete-after (and some other options) then rsync has to check every file before it can do any work, whereas normally it will find a few files to work on and start work, meanwhile incrementally scanning for more. Each file that it's working on uses 100 bytes of memory, so that's about 100MB for each million files. Not a huge amount of memory either way so probably won't cause an issue. It would be the thing about not starting any work until having fully scanned the file tree. It's all covered in the man page where it talks about those options. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: rsync --delete vs rsync --delete-after
On Thu, 2024-01-18 at 13:09 +, Michael Kjörling wrote: > > Definitely agree that a solid backup regimen (including regular > automated backups; at least one off-site copy _at least_ of critical, > hot data; and planning for the contingency that you need to restore > that backup onto a brand new system without access to anything on > your > current system -- think "home burns down at night" or "burglar" > scenario) is the _first_ step, and one that a great deal of people > still fail at. Absolutely. I use a Raspberry Pi with an external USB drive for my off-site backups with Resitic. Seems to work fine for now, draws very little power, and the 4TB of a small 2.5" disk is plenty for my personal backups, when deduplicated. Still this setup probably is too complicated for many home users, where a cloud backup or similar makes more sense. > RAID is for uptime. If a week-long outage (to get replacement > hardware > and restore the most recent backup) and a day's worth of data loss is > largely inconsequential, as quite frankly it likely is for most home > users save for the cost of replacement hardware, For me the calculation is more or less "next workday to go to the local shop for a replacement hard drive" and a few hours to restore backups. Yes, if you depend on mail order, one week might be more realistic. Then I probably would keep a spare drive around even as a home user. > that's a very > different scenario from if that same outage costs $$€€¥¥ and could > destroy your livelihood; and consequently the choices made _should_ > likely be different. Of course. As soon as you have to pay several people's salaries needlessly while they sit around for access to their data, RAID makes more sense quickly. Still, it makes sense to think about what you can do yourself vs. what needs external work done, also because somebody external to repair a RAID might not show up all that quickly unless you've got some pre-negotiated contract. > _Mirrored backups_ makes very little sense to me. If a storage device > used for storage of backups fails prematurely, just toss it and get a > new one and make a new backup. Absolutely! Just make more backups, or more backups with different, independent strategies. As much as possible I try to do two independent systems (e.g. Restic doing time-based offsite backups, and a cron job doing a simple tar.gz file into some local drive or storage). /ralph
Re: rsync --delete vs rsync --delete-after
On 18 Jan 2024 13:26 +0100, from r...@h5.or.at (Ralph Aichinger): > As a home/SOHO user, I'd rather have a working backup every few hours > or every day than some RAID10 wonder Definitely agree that a solid backup regimen (including regular automated backups; at least one off-site copy _at least_ of critical, hot data; and planning for the contingency that you need to restore that backup onto a brand new system without access to anything on your current system -- think "home burns down at night" or "burglar" scenario) is the _first_ step, and one that a great deal of people still fail at. RAID is for uptime. If a week-long outage (to get replacement hardware and restore the most recent backup) and a day's worth of data loss is largely inconsequential, as quite frankly it likely is for most home users save for the cost of replacement hardware, that's a very different scenario from if that same outage costs $$€€¥¥ and could destroy your livelihood; and consequently the choices made _should_ likely be different. _Mirrored backups_ makes very little sense to me. If a storage device used for storage of backups fails prematurely, just toss it and get a new one and make a new backup. If the backup software goes haywire and starts overwriting everything with random garbage, having the garbage mirrored isn't going to help you. It's much better to have two independent backup targets and switching between them, and figuring the switching interval into your RPO. The only time when something like mirrored backups will help you is when you have only one backup set, the backup itself works fine, but a backup drive fails, _and_ the source fails before you've been able to make a new backup. That's a _very_ narrow scenario and easily solved by having two backup sets. -- Michael Kjörling 🔗 https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”
Re: rsync --delete vs rsync --delete-after
On 2024-01-17, Default User wrote: > BTW(2), I do use rsnapshot with cron jobs to back up the internal SSD > to the primary backup drive daily (and weekly, monthly, yearly). But I > am not sure if I could also use it to do copies of the primary backup > drive to the secondary backup drive (maybe using an additional > configuration file)? You can safely rsync entire rsnapshot backups if you use --hard-links option to preserve links set by rsnapshot. You have to rsync "entire" rsnapshot directory else it can't set correct hard links.
Re: rsync --delete vs rsync --delete-after
Hello fellow Debian users, On Thu, 2024-01-18 at 12:18 +0100, hw wrote: > Always use an UPS. Here I have a somewhat contrarian view, I hope not to offend too much: For countries with stable electricity supplies (like Austria where I live) having a small UPS might actually lead to more problems instead of less, unless you are putting a lot of effort into it. Very often have I had problems with UPSes, e.g. batteries dying, the UPS going into some self test mode and inadvertedly shutting down, etc. I've had no external power outage in the last 5 or 10 years, but a UPS often needs at least one battery replacement during that time. Unless you have some sort of professional server rack and redundant 2 phase supply, in my opinion UPS make very little sense to the home or small office user. Also modern Linux systems with journalling filesystems will survive the occasional hard shutdown. Yes, I have pulled the plug out of running Linux boxes occasionally because I was too lazy to shut it down correctly and never had one break beyond the usual fsck on boot. > Always use redundancy to store data for a running system, like some > form of RAID. It won't hurt to use RAID for backups as well, though > I don't think that's required when you use it for the data you're > backing up. Here I also doubt if this is a wise suggestion for the typical home or small office user. RAID leads to lots and lots of complexity, that is often not needed in a home setup. I'd rather have a working backup setup with many independent copies before I even start thinking about RAID. Yes, disks can fail, but data loss often is due to user error and malware. RAID helps very little with the latter two causes of data loss. And all too often have I seen people mess up their complicated RAID setups, because they pulled the wrong disk when another one broke, or because they misinterpreted complicated error messages, creating unnecessary data loss out of user error by themselves. As a home/SOHO user, I'd rather have a working backup every few hours or every day than some RAID10 wonder that makes me lose more time on reading RAID documentation, and ordering spare drives (you've got one of those spares for each array, do you?) than is actually lost by not being able to restore to the exact last minute before a hard disk died. /ralph -- no UPS at home, using RAID1 md mirroring though
Re: rsync --delete vs rsync --delete-after
On 18 Jan 2024 12:15 +0100, from to...@tuxteam.de: >> **That "primary backup drive" is not a backup at all.** > > It is: against the situation you fat-finger something and react > before the next backup happens (this is a threat worth being taken > into account). For that case, a backup with more than one "back" > version would be more useful (see --link-dest in rsync for that). > > It is not against the system running amok and thrashing all attached > disks (be it by accident -- improbable, or by malice -- more probable. > cf. crypto-malware). Disconnected back ups are better here. > > It is not for against your shed going up in flames. Off site for this. OP has specified (17 Jan 19:52 UTC) that the threat model includes, among many other things, "mechanical failure" and "lightning". A single copy offers zero protection against that. -- Michael Kjörling 🔗 https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”
Re: rsync --delete vs rsync --delete-after
On Wed, 2024-01-17 at 14:52 -0500, Default User wrote: > On Wed, 2024-01-17 at 10:29 -0800, Kushal Kumaran wrote: > > On Wed, Jan 17 2024 at 11:19:39 AM, Default User > > wrote: > > > Hello! > > > > > > Opinions, please. > > > > > > I use rsync to copy my primary backup drive to a secondary backup > > > drive > > > , so that the secondary backup drive is theoretically always an > > > exact > > > copy of the primary backup drive. > > > > > > Here is the rsync command I use: > > > > > > time sudo rsync -aAXHxvv --delete-after --numeric-ids -- > > > info=progress2,stats2,name2 -- > > > exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/m > > > edia > > > /*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/ > > > > > > Question: > > > I use rsync --delete-after because it might seem to be "safer", so > > > in > > > case of a "glitch" of any kind, no file ever disappears from both > > > the > > > source drive and the destination drive. > > > > > > > What do you mean by "glitch"? Irrespective of whether you use -- > > delete > > or --delete-after, deleted files on the source are deleted on the > > destination once your rsync is complete (which is what I'd assume you > > want when you want an exact copy). I'd presume if you're ok with > > that, > > you are also fine with the deletion happening earlier in the rsync > > process? > > > > If you're concerned about accidental deletions, you should just not > > use > > any of the `--delete*` options (and give up on the exact copy > > requirement). You can look at alternatives to bare rsync that keep > > track of multiple backed-up images (rsnapshot is a very simple > > wrapper > > over rsync that can do this, for example). > > > > > However, I have read that using rsync --delete instead of rsync -- > > > delete-after is faster and uses less memory, and so is more > > > efficient. > > > > > > Note: The current copy process time varies, but takes a long time - > > > last night 131 minutes. > > > :( > > > > You can try using --delete for a couple of runs and see if it > > actually > > affects performance in your situation. > > > > > > > > Disk space used is not currently an issue. > > > > > > But, is rsync --delete AS SAFE as rsync --delete-after? > > > > You'll need to define what safety means for you. > > > > > > Hi, Kushal! > Thanks for replying. > > By "glitch", I mean anything that could interfere with the rsync copy > process. Possible causes: > - electrical outages, voltage spikes, voltage drops, "brownouts" > - mechanical failure > - earthquake > - lightning > - cat walking on keyboard > - out of memory errors > - out of disk space errors > - PEBKAC errors > - etc. > > By safe, may I try to explain using a story? > > I once read that centuries ago, a king wanted his crown to be safe. So > four guards watched his crown at all times. The guards were not > allowed to take their eves off of the crown, even for a second, until > the the their replacements, the next group of guards, all said "I see > the crown". > > I am sorry if I can not come up with a better, technical explanation. > I use this story because it has always been meaningful to me, and seems > to point to the essence of what I am getting at. > > I am writing as someone who has lost data more than once over time, for > various reasons. The loss has ranged from slightly annoying, to soul- > rending catastrophe. It is NEVER appreciated. > > I do intend to try doing rsync --delete instead of rsync --delete > after, to see if it "seems to work". > > But I just wanted to ask first ask, as it would seem better to hear > someone say "How stupid are you? I can't believe you were going to do > that!", than to have them say "How stupid are you? I can't believe you > did that!" > > On Wed, 2024-01-17 at 14:52 -0500, Default User wrote: > [...] > By "glitch", I mean anything that could interfere with the rsync copy > process. Possible causes: > - electrical outages, voltage spikes, voltage drops, "brownouts" > - mechanical failure > - earthquake > - lightning > - cat walking on keyboard > - out of memory errors > - out of disk space errors > - PEBKAC errors > - etc. > > By safe, may I try to explain using a story? > > I once read that centuries ago, a king wanted his crown to be safe. So > four guards watched his crown at all times. The guards were not > allowed to take their eves off of the crown, even for a second, until > the the their replacements, the next group of guards, all said "I see > the crown". > > I am sorry if I can not come up with a better, technical explanation. > I use this story because it has always been meaningful to me, and seems > to point to the essence of what I am getting at. > > I am writing as someone who has lost data more than once over time, for > various reasons. The loss has ranged from slightly annoying, to soul- > rending catastrophe. It is NEVER appreciated. > > I do intend to try do
Re: rsync --delete vs rsync --delete-after
On Thu, Jan 18, 2024 at 11:05:01AM +, Michael Kjörling wrote: > On 17 Jan 2024 20:23 -0500, from hunguponcont...@gmail.com (Default User): [...] > Hold on. Let's pause right here. > > **That "primary backup drive" is not a backup at all.** It is: against the situation you fat-finger something and react before the next backup happens (this is a threat worth being taken into account). For that case, a backup with more than one "back" version would be more useful (see --link-dest in rsync for that). It is not against the system running amok and thrashing all attached disks (be it by accident -- improbable, or by malice -- more probable. cf. crypto-malware). Disconnected back ups are better here. It is not for against your shed going up in flames. Off site for this. Cheers -- t signature.asc Description: PGP signature
Re: rsync --delete vs rsync --delete-after
On 17 Jan 2024 20:23 -0500, from hunguponcont...@gmail.com (Default User): > BTW, the two backup drives are external 4 Gb USB HDDs. The secondary > backup drive is always kept away from the computer, in a locked steel > box, except when it is attached to the computer to have the primary > backup drive copied to it. > > The primary backup drive is almost always attached to the computer, so > that I can access files archived there, that are not on the computer. > Probably not good practice, but that's why I have the secondary backup > drive. Hold on. Let's pause right here. **That "primary backup drive" is not a backup at all.** If at any point it can legitimately contain the only copy of a file that you want to keep, then conceptually it is not a backup. If it is continuously accessible from the system that you're trying to protect, then anything bad which happens to that system is liable to also at least potentially affect your "primary backup drive" as well. There is nothing wrong with using external storage media to expand the storage capacity of your computer, but you really shouldn't treat or consider it as being a backup, because a backup is a _second_ copy to be used if the primary copy somehow becomes unusable, corrupt, inaccessible or whatever else might be the case. Solve the right problem. > I guess in the back of my mind I was thinking of a scenario where a > file on the primary backup drive might be corrupted or deleted before > being copied to the secondary backup drive. Then if it is not present > on the primary backup drive, rsync dutifully deletes it from the > secondary backup drive. If the file is no longer on the computer's > internal SSD, I am then SOL. That is why a backup scheme should include some form of retention; ideally immutable. A single botched backup run should not risk the integrity of an only backup. > BTW(2), I do use rsnapshot with cron jobs to back up the internal SSD > to the primary backup drive daily (and weekly, monthly, yearly). But I > am not sure if I could also use it to do copies of the primary backup > drive to the secondary backup drive (maybe using an additional > configuration file)? You can use rsnapshot with any number of different configuration files. Just pass `-c` to it to name the configuration file other than the default. (For a long time I had two, and a script selecting either one based on the backup target drive in use for that particular backup. Worked great.) > As for ZFS . . . I wish! But I think the resource requirements would > be too high for my setup, so it probably would be impractical - or > impossible. And then there's the complexity. And the learning curve. ZFS works fine on low-spec'd systems. I use it on a VPS with 2 GB RAM without any problems. You really want 64-bit, but that's about it. And while ZFS _can_ be complex, and supports rather complex usage scenarios (because it is, at its core, an enterprise solution), at the basic level the biggest difference is that you write "zpool create" instead of "mkfs.ext4". -- Michael Kjörling 🔗 https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”
Re: rsync --delete vs rsync --delete-after
On 1/17/24 17:23, Default User wrote: On Wed, 2024-01-17 at 09:19 -0800, David Christensen wrote: On 1/17/24 08:19, Default User wrote: Opinions, please. ... Hi guys, thanks for the replies. YW. :-) BTW, the two backup drives are external 4 Gb USB HDDs. The secondary backup drive is always kept away from the computer, in a locked steel box, except when it is attached to the computer to have the primary backup drive copied to it. That means the live data and all the backups are exposed to the same computer if and when it is compromised. It would be safer to do the copy on another computer. If you only have one computer, then boot live media to do the copy. The primary backup drive is almost always attached to the computer, so that I can access files archived there, that are not on the computer. Accidental file modification and deletion are probably the most common failure modes. One of the features of ZFS is snapshots. Snapshots can be accessed via the hidden ".zfs/snapshot/" directory in the root of each file system. Users can browse the snapshots and copy out whatever files and directories they need, subject to the file system permissions at the time the snapshot was taken. So, recovery from the above failure is "self serve" -- no sysadmin required, no backup/ restore software required. Probably not good practice, but that's why I have the secondary backup drive. I guess in the back of my mind I was thinking of a scenario where a file on the primary backup drive might be corrupted or deleted before being copied to the secondary backup drive. Then if it is not present on the primary backup drive, rsync dutifully deletes it from the secondary backup drive. If the file is no longer on the computer's internal SSD, I am then SOL. ZFS snapshots are read-only. Individual files and directories within a snapshot cannot be created, updated, or deleted. Another feature of ZFS is that both data and metadata are checksummed. So, if you can read a file, the metadata and data are good. "bitrot" is extremely unlikely. I have also thought of trying to use partclone to copy the data from the primary backup drive to the secondary backup drive. Why not try, Transferring 4 TB at 150 MB/s would require: 4 TB / 150 MB/s = 26,667 s Or, about 7.4 hours. since rsync takes an hour and a half, every day! How much data is rsync(1) transferring? Another feature of ZFS snapshots is that they can be replicated from one pool to another. The first replication is full. Subsequent replications are incremental. Here are some recent transfers from my backup server to a removable HDD: frequency bytes real timebandwidth = == === = weekly 11,125,755,324 25m08.848s 7.37 MB/s monthly75,218,102,216 140m53.710s 8.90 MB/s As for ZFS . . . I wish! But I think the resource requirements would be too high for my setup, so it probably would be impractical - or impossible. ZFS will use as much or as little resources as you provide. (ZFS is known to consume most of available memory OOTB; this can be tuned. The simple answer is to put it on a dedicated file server/ NAS.) Please tell us about your setup -- hardware, stored data, and I/O workload. And then there's the complexity. I find ZFS has more conceptual integrity than cobbling together fdisk(8), mdadm(8), cryptsetup(8), lvm(8), mkfs(8), etc.. And the learning curve. The Lucas books got me up the learning curve: https://mwl.io/nonfiction/os#fmzfs https://mwl.io/nonfiction/os#fmaz While both have "FreeBSD" in the title, most of the content applies equally to OpenZFS on Debian. Finally I really should have a third backup drive in the mix. Yes, I am familiar with the 1-2-3 backup theory. But the third backup drive could not be off-site, for various reasons. And I do have other things to do rather than spend all day, every day, managing backups. Disaster preparedness is an open-ended problem. Only you can decide how much solution to give it. David
Re: rsync --delete vs rsync --delete-after
> However, I have read that using rsync --delete instead of rsync -- > delete-after is faster and uses less memory, and so is more efficient. I'd be surprised if it makes a significant difference. Stefan
Re: rsync --delete vs rsync --delete-after
On 2024-01-17, Default User wrote: > By "glitch", I mean anything that could interfere with the rsync copy > process. Possible causes: Whatever the cause you just have to get return code and restart rsync until it complete succesfully. Then you are sure to have an exact copy. To cope with errors in primary backup you must have multiple copies as suggest Michael. Personnally I use rsnapshot (a tool using rsync itself) for that.
Re: rsync --delete vs rsync --delete-after
On Wed, 2024-01-17 at 09:19 -0800, David Christensen wrote: > On 1/17/24 08:19, Default User wrote: > > Hello! > > > > Opinions, please. > > > > I use rsync to copy my primary backup drive to a secondary backup > > drive > > , so that the secondary backup drive is theoretically always an > > exact > > copy of the primary backup drive. > > > > Here is the rsync command I use: > > > > time sudo rsync -aAXHxvv --delete-after --numeric-ids -- > > info=progress2,stats2,name2 -- > > exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/m > > edia > > /*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/ > > > > Question: > > I use rsync --delete-after because it might seem to be "safer", so > > in > > case of a "glitch" of any kind, no file ever disappears from both > > the > > source drive and the destination drive. > > > > However, I have read that using rsync --delete instead of rsync -- > > delete-after is faster and uses less memory, and so is more > > efficient. > > > > Note: The current copy process time varies, but takes a long time - > > last night 131 minutes. > > :( > > > > Disk space used is not currently an issue. > > > > But, is rsync --delete AS SAFE as rsync --delete-after? > > > In the past, I used the --backup and --backup-dir options to retain > files on the destination. > > > Then I moved my primary backup to ZFS, implemented snapshots, and > implemented replication to the secondard backup devices. > > > David > Hi guys, thanks for the replies. BTW, the two backup drives are external 4 Gb USB HDDs. The secondary backup drive is always kept away from the computer, in a locked steel box, except when it is attached to the computer to have the primary backup drive copied to it. The primary backup drive is almost always attached to the computer, so that I can access files archived there, that are not on the computer. Probably not good practice, but that's why I have the secondary backup drive. I guess in the back of my mind I was thinking of a scenario where a file on the primary backup drive might be corrupted or deleted before being copied to the secondary backup drive. Then if it is not present on the primary backup drive, rsync dutifully deletes it from the secondary backup drive. If the file is no longer on the computer's internal SSD, I am then SOL. BTW(2), I do use rsnapshot with cron jobs to back up the internal SSD to the primary backup drive daily (and weekly, monthly, yearly). But I am not sure if I could also use it to do copies of the primary backup drive to the secondary backup drive (maybe using an additional configuration file)? I have also thought of trying to use partclone to copy the data from the primary backup drive to the secondary backup drive. Why not try, since rsync takes an hour and a half, every day! As for ZFS . . . I wish! But I think the resource requirements would be too high for my setup, so it probably would be impractical - or impossible. And then there's the complexity. And the learning curve. Finally I really should have a third backup drive in the mix. Yes, I am familiar with the 1-2-3 backup theory. But the third backup drive could not be off-site, for various reasons. And I do have other things to do rather than spend all day, every day, managing backups.
Re: rsync --delete vs rsync --delete-after
On 18/1/24 04:19, David Christensen wrote: > I use rsync to copy my primary backup drive to a secondary backup drive Good morning I wonder why both processes don't copy from the original data; so that you don't copy a potential glitch in the first backup? on a separate matter Glitch? Power goes off inconveniently? -- All the best Keith Bainbridge keithr...@gmail.com keith.bainbridge.3...@gmail.com +61 (0)447 667 468 UTC + 10:00
Re: rsync --delete vs rsync --delete-after
Hi, On Wed, Jan 17, 2024 at 02:52:49PM -0500, Default User wrote: > By "glitch", I mean anything that could interfere with the rsync copy > process. Possible causes: > - electrical outages, voltage spikes, voltage drops, "brownouts" > - mechanical failure > - earthquake > - lightning > - cat walking on keyboard > - out of memory errors > - out of disk space errors > - PEBKAC errors > - etc. But, both --delete and --delete-after only delete things from the destination that ALREADY are missing on the source, so in which of the above situations would something that still exists somewhere be accidentally deleted with either option? In my view the only ones that apply would be human error / cat standard behaviour but even then you'd have to notice it had happened and abort the transfer before rsync has chance to do the delete. Do you typically sit and watch these transfers? It doesn't sound like any kind of backup to me if it doesn't have history, i.e. the state of your data *before* the cat walked on 'r' 'm' ' ' '-' 'f' 'r' ' ' '.' ''. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: rsync --delete vs rsync --delete-after
On 17 Jan 2024 14:52 -0500, from hunguponcont...@gmail.com (Default User): > I am writing as someone who has lost data more than once over time, for > various reasons. The loss has ranged from slightly annoying, to soul- > rending catastrophe. It is NEVER appreciated. I think this gets closer to the root of what you're trying to achieve: it sounds to me as though no matter what happens, you want to have a restorable backup which you can trust to (reasonably well) match the state of the source at the time when that backup was taken. That is a commendable goal. Twiddling with options to rsync won't offer you that in light of several of the scenarios you listed, though; not least a lightning strike. A lightning strike will blow out the drive just as much no matter what options you're using to invoke rsync. What you need is really rather multiple copies. I suggest to get a second drive to use for backup purposes along with the one you currently have. Only ever connect one of them at the same time to data or power anywhere. Ideally, always keep at least one of them in a separate location, powered off and disconnected. That way, if you mess up one, or if one of them fails (for any reason), or whatever else might happen, you have the other. It might not be quite as up to date, but it will be in a known-good state. For less drastic failures, really, look at rsnapshot. It's a wrapper around rsync which makes maintaining multiple revisions of a backup much easier. (It essentially passes to rsync with --link-dest, and manages the respective root directories.) -- Michael Kjörling 🔗 https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”
Re: rsync --delete vs rsync --delete-after
On Wed, 2024-01-17 at 10:29 -0800, Kushal Kumaran wrote: > On Wed, Jan 17 2024 at 11:19:39 AM, Default User > wrote: > > Hello! > > > > Opinions, please. > > > > I use rsync to copy my primary backup drive to a secondary backup > > drive > > , so that the secondary backup drive is theoretically always an > > exact > > copy of the primary backup drive. > > > > Here is the rsync command I use: > > > > time sudo rsync -aAXHxvv --delete-after --numeric-ids -- > > info=progress2,stats2,name2 -- > > exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/m > > edia > > /*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/ > > > > Question: > > I use rsync --delete-after because it might seem to be "safer", so > > in > > case of a "glitch" of any kind, no file ever disappears from both > > the > > source drive and the destination drive. > > > > What do you mean by "glitch"? Irrespective of whether you use -- > delete > or --delete-after, deleted files on the source are deleted on the > destination once your rsync is complete (which is what I'd assume you > want when you want an exact copy). I'd presume if you're ok with > that, > you are also fine with the deletion happening earlier in the rsync > process? > > If you're concerned about accidental deletions, you should just not > use > any of the `--delete*` options (and give up on the exact copy > requirement). You can look at alternatives to bare rsync that keep > track of multiple backed-up images (rsnapshot is a very simple > wrapper > over rsync that can do this, for example). > > > However, I have read that using rsync --delete instead of rsync -- > > delete-after is faster and uses less memory, and so is more > > efficient. > > > > Note: The current copy process time varies, but takes a long time - > > last night 131 minutes. > > :( > > You can try using --delete for a couple of runs and see if it > actually > affects performance in your situation. > > > > > Disk space used is not currently an issue. > > > > But, is rsync --delete AS SAFE as rsync --delete-after? > > You'll need to define what safety means for you. > Hi, Kushal! Thanks for replying. By "glitch", I mean anything that could interfere with the rsync copy process. Possible causes: - electrical outages, voltage spikes, voltage drops, "brownouts" - mechanical failure - earthquake - lightning - cat walking on keyboard - out of memory errors - out of disk space errors - PEBKAC errors - etc. By safe, may I try to explain using a story? I once read that centuries ago, a king wanted his crown to be safe. So four guards watched his crown at all times. The guards were not allowed to take their eves off of the crown, even for a second, until the the their replacements, the next group of guards, all said "I see the crown". I am sorry if I can not come up with a better, technical explanation. I use this story because it has always been meaningful to me, and seems to point to the essence of what I am getting at. I am writing as someone who has lost data more than once over time, for various reasons. The loss has ranged from slightly annoying, to soul- rending catastrophe. It is NEVER appreciated. I do intend to try doing rsync --delete instead of rsync --delete after, to see if it "seems to work". But I just wanted to ask first ask, as it would seem better to hear someone say "How stupid are you? I can't believe you were going to do that!", than to have them say "How stupid are you? I can't believe you did that!"
Re: rsync --delete vs rsync --delete-after
On Wed, Jan 17 2024 at 11:19:39 AM, Default User wrote: > Hello! > > Opinions, please. > > I use rsync to copy my primary backup drive to a secondary backup drive > , so that the secondary backup drive is theoretically always an exact > copy of the primary backup drive. > > Here is the rsync command I use: > > time sudo rsync -aAXHxvv --delete-after --numeric-ids -- > info=progress2,stats2,name2 -- > exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media > /*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/ > > Question: > I use rsync --delete-after because it might seem to be "safer", so in > case of a "glitch" of any kind, no file ever disappears from both the > source drive and the destination drive. > What do you mean by "glitch"? Irrespective of whether you use --delete or --delete-after, deleted files on the source are deleted on the destination once your rsync is complete (which is what I'd assume you want when you want an exact copy). I'd presume if you're ok with that, you are also fine with the deletion happening earlier in the rsync process? If you're concerned about accidental deletions, you should just not use any of the `--delete*` options (and give up on the exact copy requirement). You can look at alternatives to bare rsync that keep track of multiple backed-up images (rsnapshot is a very simple wrapper over rsync that can do this, for example). > However, I have read that using rsync --delete instead of rsync -- > delete-after is faster and uses less memory, and so is more efficient. > > Note: The current copy process time varies, but takes a long time - > last night 131 minutes. > :( You can try using --delete for a couple of runs and see if it actually affects performance in your situation. > > Disk space used is not currently an issue. > > But, is rsync --delete AS SAFE as rsync --delete-after? You'll need to define what safety means for you. -- regards, kushal
Re: rsync --delete vs rsync --delete-after
On 1/17/24 08:19, Default User wrote: Hello! Opinions, please. I use rsync to copy my primary backup drive to a secondary backup drive , so that the secondary backup drive is theoretically always an exact copy of the primary backup drive. Here is the rsync command I use: time sudo rsync -aAXHxvv --delete-after --numeric-ids -- info=progress2,stats2,name2 -- exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media /*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/ Question: I use rsync --delete-after because it might seem to be "safer", so in case of a "glitch" of any kind, no file ever disappears from both the source drive and the destination drive. However, I have read that using rsync --delete instead of rsync -- delete-after is faster and uses less memory, and so is more efficient. Note: The current copy process time varies, but takes a long time - last night 131 minutes. :( Disk space used is not currently an issue. But, is rsync --delete AS SAFE as rsync --delete-after? In the past, I used the --backup and --backup-dir options to retain files on the destination. Then I moved my primary backup to ZFS, implemented snapshots, and implemented replication to the secondard backup devices. David
rsync --delete vs rsync --delete-after
Hello! Opinions, please. I use rsync to copy my primary backup drive to a secondary backup drive , so that the secondary backup drive is theoretically always an exact copy of the primary backup drive. Here is the rsync command I use: time sudo rsync -aAXHxvv --delete-after --numeric-ids -- info=progress2,stats2,name2 -- exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media /*","/lost+found"} /media/default/MSD0001/ /media/default/MSD0002/ Question: I use rsync --delete-after because it might seem to be "safer", so in case of a "glitch" of any kind, no file ever disappears from both the source drive and the destination drive. However, I have read that using rsync --delete instead of rsync -- delete-after is faster and uses less memory, and so is more efficient. Note: The current copy process time varies, but takes a long time - last night 131 minutes. :( Disk space used is not currently an issue. But, is rsync --delete AS SAFE as rsync --delete-after?