-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 ZFS does have big RAM requirements. 8GB of RAM is pretty much the minimum. As for CPU besides being new enough to be on a motherboard with 8GB of RAM you should be fine.
On 04/06/2015 12:25 PM, Clint Olsen wrote: > Not to mention the fact that ZFS requires considerable hardware > resources (CPU & memory) to perform well. It also requires you to > learn a whole new terminology to wrap your head around it. > > It's certainly not a trivial swap to say the least... > > Thanks, > > -Clint > > On Mon, Apr 6, 2015 at 9:12 AM, Ken Chase > <rsync-list-m...@sizone.org <mailto:rsync-list-m...@sizone.org>> > wrote: > > This has been a consideration. But it pains me that a tiny > change/addition to the rsync option set would save much time and > space for other legit use cases. > > We know rsync very well, we dont know ZFS very well (licensing kept > the tech out of our linux-centric operations). We've been using it > but we're not experts yet. > > Thanks for the suggestion. > > /kc > > On Mon, Apr 06, 2015 at 12:07:05PM -0400, Kevin Korb said: Since > you are in an environment with millions of files I highly recommend > that you move to ZFS storage and use ZFS's subvolume snapshots > instead of --link-dest. It is much more space efficient, rsync run > time efficient, and the old backups can be deleted in seconds. > Rsync doesn't have to understand anything about ZFS. You just > rsync to the same directory every time and have ZFS do a >> snapshot > on that directory between runs. > > On 04/06/2015 01:51 AM, Ken Chase wrote: >> Feature request: allow --link-dest dir to be linked to even if >> file exists in target. > >> This statement from the man page is adhered to too strongly >> IMHO: > >> "This option works best when copying into an empty destination >> hierarchy, as rsync treats existing files as definitive (so it >> never looks in the link-dest dirs when a destination file >> already exists)". > >> I was suprised by this behaviour as generally the scheme is to >> be efficient/save space with rsync. > >> When the file is out of date but exists in the --l-d target, it >> would be great if it could be removed and linked. If an option >> was supplied to request this behaviour, I'd actually throw some >> money at making it happen. (And a further option to retain a >> copy if inode permissions/ownership would otherwise be changed.) > >> Reasoning: > >> I backup many servers with --link-dest that have filesystems of >> 10+M files on them. I do not delete old backups - which take >> 60min per tree or more just so rsync can recreate them all in an >> empty target dir when <1% of files change per day (takes 3-5 hrs >> per backup!). > >> Instead, I cycle them in with mv $olddate $today then rsync >> --del --link-dest over them - takes 30-60 min depending. (Yes, >> some malleability of permissions risk there, mostly interested >> in contents tho). Problem is, if a file exists AT ALL, even out >> of date, a new copy is put overtop of it per the above man page >> decree. > >> Thus much more disk space is used. Running this scheme with >> moving old backups to be written overtop of accumulates many >> copies of the exact same file over time. Running pax -rpl over >> the copies before rsyncing to them works (and saves much space!), >> but takes a very long time as it traverses and compares 2 large >> backup trees thrashing the same device (in the order of 3-5x the >> rsync's time, 3-5 hrs for pax - hardlink(1) is far worse, I >> suspect a some non-linear algorithm therein - it ran 3-5x slower >> than pax again). > >> I have detailed an example of this scenario at > > >> http://unix.stackexchange.com/questions/193308/rsyncs-link-dest-option-does-not-link-identical-files-if-an-old-file-exists > >> which also indicates --delete-before and --whole-file do not >> help at all. > >> /kc > > >> -- Please use reply-all for most replies to avoid omitting the > mailing list. >> To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync >> Before posting, read: > http://www.catb.org/~esr/faqs/smart-questions.html > > -- Ken Chase - ken att heavycomputing.ca > <http://heavycomputing.ca> Toronto Canada Heavy Computing - Clued > bandwidth, colocation and managed linux VPS @151 Front St. W. -- > Please use reply-all for most replies to avoid omitting the > mailing list. To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync Before posting, > read: http://www.catb.org/~esr/faqs/smart-questions.html > > > > - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Florida k...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlUitMsACgkQVKC1jlbQAQcNBgCeLznsYokPy4A3BGmsRmabFmag C4IAoKWUVb+azUEXtMFdUQHKUTU4kV3+ =cuLG -----END PGP SIGNATURE----- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html