Re: [GTALUG] On the subject of backups.
I haven't used Gluster personally, but have you tried turning performance.parallel-readdir on? https://docs.gluster.org/en/latest/release-notes/3.10.0/#implemented-parallel-readdirp-with-distribute-xlator It seems there's a reason why it's on by default ( https://www.spinics.net/lists/gluster-devel/msg25518.html) but maybe it'd still be worth it for you? On Mon, May 4, 2020 at 9:55 AM Alvin Starr via talk wrote: > > I am hoping someone has seen this kind of problem before and knows of a > solution. > I have a client who has file systems filled with lots of small files on > the orders of hundreds of millions of files. > Running something like a find on filesystem takes the better part of a > week so any kind of directory walking backup tool will take even longer > to run. > The actual data-size for 100M files is on the order of 15TB so there is > a lot of data to backup but the data only increases on the order of tens > to hundreds of MB a day. > > > Even things like xfsdump take a long time. > For example I tried xfsdump on a 50M file set and it took over 2 days to > complete. > > The only thing that seems to be workable is Veeam. > It will run an incremental volume snapshot in a few hours a night but I > dislike adding proprietary kernel modules into the systems. > > > -- > Alvin Starr || land: (647)478-6285 > Netvel Inc. || Cell: (416)806-0133 > al...@netvel.net || > > --- > Post to this mailing list talk@gtalug.org > Unsubscribe from this mailing list > https://gtalug.org/mailman/listinfo/talk > --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On Wed, 2020/05/06 10:38:29AM -0400, Howard Gibson via talk wrote: | > ZFS is another option. And it handles delta-backups very easily. | |How do you recover stuff from delta backups? You have to figure which backup the file or directory is in, right? Remember that snapshots, like RAID, are not actually backups, unless they are on a different machine, in a different place. ZFS makes it easy: You can browse through snapshots for /mypool/myfs by looking in /mypool/myfs/.zfs/snapshot and if your ZFS snapshots are named using dates, easy peasy to choose when. You can also brute force and find /mypool/myfs/.zfs/snapshot -name 'myfile.tex' -ls and find what's there. You can use "zfs rollback" to revert to a snapshot. You can use "zfs send ... | zfs recv ..." to copy a specific snapshot (or group of snapshots) to another pool, system, etc. And of course, when you create a snapshot, you could create your own index listing of what's there for easy grepping. ZFS is great. You can still (and likely should) continue to backup to Blu-ray, but ZFS will make sure your files don't rot in place unnoticed. Hope that helps, cheers John --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On 5/6/20 10:18 AM, Lennart Sorensen via talk wrote: On Wed, May 06, 2020 at 07:25:29AM -0400, David Mason wrote: ZFS is another option. And it handles delta-backups very easily. It is however not the XFS that glusterfs says to use. If glusterfs is involved, XFS really seems to be the only option. There is Gluster hosted documentation on how to use ZFS with Gluster but it not the preferred option. Putting Gluster on a thinlvm is the Gluster way for doing snapshots. I have found a package called wyng-backup(https://github.com/tasket/wyng-backup). This may solve my problem but it would require me to change all my volumes to thin provisioning. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On Wed, 6 May 2020 07:25:29 -0400 David Mason via talk wrote: > ZFS is another option. And it handles delta-backups very easily. David, How do you recover stuff from delta backups? You have to figure which backup the file or directory is in, right? My backup recoveries, admittedly here at home, consist of me recovering files I stupidly modified incorrectly, and some accidentally deleted directories. In the case of the directories, I noticed the problem several years after I did it. Since I back up everything, all I had to do was pull out the backup Blu-ray from shortly after I worked on the directory. -- Howard Gibson hgib...@eol.ca jhowardgib...@gmail.com http://home.eol.ca/~hgibson --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On Wed, May 06, 2020 at 07:25:29AM -0400, David Mason wrote: > ZFS is another option. And it handles delta-backups very easily. It is however not the XFS that glusterfs says to use. If glusterfs is involved, XFS really seems to be the only option. -- Len Sorensen --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On 5/6/20 7:25 AM, David Mason via talk wrote: ZFS is another option. And it handles delta-backups very easily. ../Dave I have been following ZFS on and off and it looks interesting but I am kind of stuck because it is not included in Centos/RH which is a requirement in this case. It would be interesting to test it at scale just to see how well is does work. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
ZFS is another option. And it handles delta-backups very easily. ../Dave On May 5, 2020, 11:27 PM -0400, Lennart Sorensen via talk , wrote: > On Mon, May 04, 2020 at 10:42:25PM -0400, Alvin Starr via talk wrote: > > The files are generally a few hundred KB each. They may run into a few MB > > but that's about it. > > I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of > > favor after the lead developer got sent away for murder. > > Reiser was much faster and more reliable than ext at the time. > > It would actually be interesting to see if running a reiserfs or btrfs > > filesystem would actually make a significant difference but in the long run > > I am kind of stuck with Centos/RH supported file systems and reiser and > > btrfs are not part of that mix anymore. > > ReiserFS was not reliable. I certainly stopped using it long before > the developer issues happened. The silent file content loss was just > unacceptable. And it wasn't a rare occurance. I saw it on many systems > many times. ext2 and 3 you could at least trust with your data even if > they were quite a bit slower. Certainly these days RHEL supports ext2/3/4 > and XFS (their default and preferred). I use ext4 because it works well. > GlusterFS defaults to XFS and while technically it can use other > filesystems (and many people do run ext4 on it apparently) I don't > believe they support that setup. > > > I am not sure how much I can get by tweaking the filesystem. > > I would need to get a 50x -100x improvement to make backups complete in a > > few hours. > > Most stuff I have read comparing various filesystems and performance are > > talking about percentage differences that is much less than 100%. > > > > I have a feeling that the only answer will be something like Veeam where > > only changed blocks are backed up. > > A directory tree walk just takes too long. > > Well, does the system have enough ram? That is something that often > isn't hard to increase. XFS has certainly in the past been known to > require a fair bit of ram to manage well. > > -- > Len Sorensen > --- > Post to this mailing list talk@gtalug.org > Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On 5/6/20 12:37 AM, Nicholas Krause wrote: [snip] Well, does the system have enough ram? That is something that often isn't hard to increase. XFS has certainly in the past been known to require a fair bit of ram to manage well. I mentioned to check /proc/meminfo as well to look at cache in case you missed that. You did mention that and it is on my plate to research the amount of ram that is suggested for this kind of application. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On 5/5/20 11:27 PM, Lennart Sorensen wrote: On Mon, May 04, 2020 at 10:42:25PM -0400, Alvin Starr via talk wrote: The files are generally a few hundred KB each. They may run into a few MB but that's about it. I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of favor after the lead developer got sent away for murder. Reiser was much faster and more reliable than ext at the time. It would actually be interesting to see if running a reiserfs or btrfs filesystem would actually make a significant difference but in the long run I am kind of stuck with Centos/RH supported file systems and reiser and btrfs are not part of that mix anymore. ReiserFS was not reliable. I certainly stopped using it long before the developer issues happened. The silent file content loss was just unacceptable. And it wasn't a rare occurance. I saw it on many systems many times. ext2 and 3 you could at least trust with your data even if they were quite a bit slower. Certainly these days RHEL supports ext2/3/4 and XFS (their default and preferred). I use ext4 because it works well. GlusterFS defaults to XFS and while technically it can use other filesystems (and many people do run ext4 on it apparently) I don't believe they support that setup. This is a nice example of YMMV. I moved my critical data from ext2/3 because they would trash my data and corrupt the whole filesystem under certain cases. I had much better luck with Reiser even in the face of a couple of system crashes. I am not sure how much I can get by tweaking the filesystem. I would need to get a 50x -100x improvement to make backups complete in a few hours. Most stuff I have read comparing various filesystems and performance are talking about percentage differences that is much less than 100%. I have a feeling that the only answer will be something like Veeam where only changed blocks are backed up. A directory tree walk just takes too long. Well, does the system have enough ram? That is something that often isn't hard to increase. XFS has certainly in the past been known to require a fair bit of ram to manage well. The system has 32G and runs nothing but gluster. Not sure if that is enough ram and will require a bit of researching. It looks like the system has about 15G of cached data. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On 5/5/20 11:27 PM, Lennart Sorensen wrote: On Mon, May 04, 2020 at 10:42:25PM -0400, Alvin Starr via talk wrote: The files are generally a few hundred KB each. They may run into a few MB but that's about it. I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of favor after the lead developer got sent away for murder. Reiser was much faster and more reliable than ext at the time. It would actually be interesting to see if running a reiserfs or btrfs filesystem would actually make a significant difference but in the long run I am kind of stuck with Centos/RH supported file systems and reiser and btrfs are not part of that mix anymore. ReiserFS was not reliable. I certainly stopped using it long before the developer issues happened. The silent file content loss was just unacceptable. And it wasn't a rare occurance. I saw it on many systems many times. ext2 and 3 you could at least trust with your data even if they were quite a bit slower. Certainly these days RHEL supports ext2/3/4 and XFS (their default and preferred). I use ext4 because it works well. GlusterFS defaults to XFS and while technically it can use other filesystems (and many people do run ext4 on it apparently) I don't believe they support that setup. I am not sure how much I can get by tweaking the filesystem. I would need to get a 50x -100x improvement to make backups complete in a few hours. Most stuff I have read comparing various filesystems and performance are talking about percentage differences that is much less than 100%. I have a feeling that the only answer will be something like Veeam where only changed blocks are backed up. A directory tree walk just takes too long. Well, does the system have enough ram? That is something that often isn't hard to increase. XFS has certainly in the past been known to require a fair bit of ram to manage well. I mentioned to check /proc/meminfo as well to look at cache in case you missed that. Nick -- Fundamentally an organism has conscious mental states if and only if there is something that it is like to be that organism--something it is like for the organism. - Thomas Nagel --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On Mon, May 04, 2020 at 10:42:25PM -0400, Alvin Starr via talk wrote: > The files are generally a few hundred KB each. They may run into a few MB > but that's about it. > I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of > favor after the lead developer got sent away for murder. > Reiser was much faster and more reliable than ext at the time. > It would actually be interesting to see if running a reiserfs or btrfs > filesystem would actually make a significant difference but in the long run > I am kind of stuck with Centos/RH supported file systems and reiser and > btrfs are not part of that mix anymore. ReiserFS was not reliable. I certainly stopped using it long before the developer issues happened. The silent file content loss was just unacceptable. And it wasn't a rare occurance. I saw it on many systems many times. ext2 and 3 you could at least trust with your data even if they were quite a bit slower. Certainly these days RHEL supports ext2/3/4 and XFS (their default and preferred). I use ext4 because it works well. GlusterFS defaults to XFS and while technically it can use other filesystems (and many people do run ext4 on it apparently) I don't believe they support that setup. > I am not sure how much I can get by tweaking the filesystem. > I would need to get a 50x -100x improvement to make backups complete in a > few hours. > Most stuff I have read comparing various filesystems and performance are > talking about percentage differences that is much less than 100%. > > I have a feeling that the only answer will be something like Veeam where > only changed blocks are backed up. > A directory tree walk just takes too long. Well, does the system have enough ram? That is something that often isn't hard to increase. XFS has certainly in the past been known to require a fair bit of ram to manage well. -- Len Sorensen --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On 5/4/20 10:42 PM, Alvin Starr wrote: On 5/4/20 7:28 PM, nick wrote: On 2020-05-04 2:12 p.m., Alvin Starr via talk wrote: On 5/4/20 1:26 PM, Lennart Sorensen via talk wrote: On Mon, May 04, 2020 at 04:38:28PM +0200, ac via talk wrote: Hi Alvin, On a 2TB dataset, with +-600k files, I have piped tree to less with limited joy, it took a few hours and at least I could search for what I was looking for... - 15TB and 100M is another animal though and as disk i/o will be your bottleneck, anything will take long, no? now, for my own info/interest, can you tell me which fs is used for this ext3? Hmm, sounds awful slow. Just for fun I ran find on one of my drives: # time find /data | wc -l 1825463 real 3m57s.208s That is with 5.3T used out of 6.0TB. Running it a second time when it is cached takes 7.7s. Tree takes 14.7s. Another volume: # time find /mythdata | wc -l 54972 real 0m1.924s That is with 15 TB out of 15 TB in use (yes that one always fills up for some reason). Both of those are lvm volumes with ext4 on top of software raid6 using 5400rpm WD red drives. Seems either XFS is unbelievable bad, or there isn't enough ram to cache the filesystem metadata if you are having a problem with 100M files. I only have a measly 32GB in my home machine. I believe the directory hierarchy has a lot to do with the performance. It seems that the listing time is non-linear although I do not believe its an N^^2 kind of problem. I would have said the same as you before I started having to deal with 10's of millions of files. The first question I would have is how big are the actual files versus space used. Most file systems try to merge files that don't allocate in a single block to save blocks and get better space usage. Assuming a) the size of the files being rather small and b) The amount, I would be curious if ReiserFS or brtfs helps either as most have much better tail merging into metadata blocks to my knowledge. Better packing the small files can help as disk seeks are a problem here it seems. My disks will solve to like 10 MB/S on ext4 with lots of small files like this due to seeks. The files are generally a few hundred KB each. They may run into a few MB but that's about it. I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of favor after the lead developer got sent away for murder. Reiser was much faster and more reliable than ext at the time. It would actually be interesting to see if running a reiserfs or btrfs filesystem would actually make a significant difference but in the long run I am kind of stuck with Centos/RH supported file systems and reiser and btrfs are not part of that mix anymore. Interesting, I recall CentOS 7 and RHEL 7 supporting it. You must be on a older system unless my memory is wrong :). The problem really is seek times. I've not sure if there is a way to tune the kernel from memory for this but that's your problem unfortunately. I am not sure how much I can get by tweaking the filesystem. I would need to get a 50x -100x improvement to make backups complete in a few hours. Most stuff I have read comparing various filesystems and performance are talking about percentage differences that is much less than 100%. I have a feeling that the only answer will be something like Veeam where only changed blocks are backed up. A directory tree walk just takes too long. See my above comments but the other two ideas would be to use some short of filesystem or LVM snapshots or try dd as a benchmark to see if a raw copy tool does better. Snapshots are probably the best way through or only changed blocked based tools as you logically came to the conclusion of. Good luck through as lots of small files are always a pain in transferring and especially on rotating disks, Nick The other question as pointed out here is how much memory the page cache and other kernel caches are using. I would check /proc/meminfo to start as this may be another logical solution already pointed out. Maybe that helps, Nick -- Fundamentally an organism has conscious mental states if and only if there is something that it is like to be that organism--something it is like for the organism. - Thomas Nagel --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On 5/4/20 7:28 PM, nick wrote: On 2020-05-04 2:12 p.m., Alvin Starr via talk wrote: On 5/4/20 1:26 PM, Lennart Sorensen via talk wrote: On Mon, May 04, 2020 at 04:38:28PM +0200, ac via talk wrote: Hi Alvin, On a 2TB dataset, with +-600k files, I have piped tree to less with limited joy, it took a few hours and at least I could search for what I was looking for... - 15TB and 100M is another animal though and as disk i/o will be your bottleneck, anything will take long, no? now, for my own info/interest, can you tell me which fs is used for this ext3? Hmm, sounds awful slow. Just for fun I ran find on one of my drives: # time find /data | wc -l 1825463 real 3m57s.208s That is with 5.3T used out of 6.0TB. Running it a second time when it is cached takes 7.7s. Tree takes 14.7s. Another volume: # time find /mythdata | wc -l 54972 real 0m1.924s That is with 15 TB out of 15 TB in use (yes that one always fills up for some reason). Both of those are lvm volumes with ext4 on top of software raid6 using 5400rpm WD red drives. Seems either XFS is unbelievable bad, or there isn't enough ram to cache the filesystem metadata if you are having a problem with 100M files. I only have a measly 32GB in my home machine. I believe the directory hierarchy has a lot to do with the performance. It seems that the listing time is non-linear although I do not believe its an N^^2 kind of problem. I would have said the same as you before I started having to deal with 10's of millions of files. The first question I would have is how big are the actual files versus space used. Most file systems try to merge files that don't allocate in a single block to save blocks and get better space usage. Assuming a) the size of the files being rather small and b) The amount, I would be curious if ReiserFS or brtfs helps either as most have much better tail merging into metadata blocks to my knowledge. Better packing the small files can help as disk seeks are a problem here it seems. My disks will solve to like 10 MB/S on ext4 with lots of small files like this due to seeks. The files are generally a few hundred KB each. They may run into a few MB but that's about it. I use to use ReiserFS back in the days of ext2/3 but it kind of fell out of favor after the lead developer got sent away for murder. Reiser was much faster and more reliable than ext at the time. It would actually be interesting to see if running a reiserfs or btrfs filesystem would actually make a significant difference but in the long run I am kind of stuck with Centos/RH supported file systems and reiser and btrfs are not part of that mix anymore. I am not sure how much I can get by tweaking the filesystem. I would need to get a 50x -100x improvement to make backups complete in a few hours. Most stuff I have read comparing various filesystems and performance are talking about percentage differences that is much less than 100%. I have a feeling that the only answer will be something like Veeam where only changed blocks are backed up. A directory tree walk just takes too long. The other question as pointed out here is how much memory the page cache and other kernel caches are using. I would check /proc/meminfo to start as this may be another logical solution already pointed out. Maybe that helps, Nick -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On 2020-05-04 2:12 p.m., Alvin Starr via talk wrote: > On 5/4/20 1:26 PM, Lennart Sorensen via talk wrote: >> On Mon, May 04, 2020 at 04:38:28PM +0200, ac via talk wrote: >>> Hi Alvin, >>> >>> On a 2TB dataset, with +-600k files, I have piped tree to less with >>> limited joy, it took a few hours and at least I could search for >>> what I was looking for... - 15TB and 100M is another animal though >>> and as disk i/o will be your bottleneck, anything will take long, no? >>> >>> now, for my own info/interest, can you tell me which fs is used for this >>> ext3? >> Hmm, sounds awful slow. >> >> Just for fun I ran find on one of my drives: >> >> # time find /data | wc -l >> 1825463 >> real 3m57s.208s >> >> That is with 5.3T used out of 6.0TB. >> >> Running it a second time when it is cached takes 7.7s. Tree takes 14.7s. >> >> Another volume: >> # time find /mythdata | wc -l >> 54972 >> >> real 0m1.924s >> >> That is with 15 TB out of 15 TB in use (yes that one always fills up >> for some reason). >> >> Both of those are lvm volumes with ext4 on top of software raid6 using >> 5400rpm WD red drives. >> >> Seems either XFS is unbelievable bad, or there isn't enough ram to cache >> the filesystem metadata if you are having a problem with 100M files. >> I only have a measly 32GB in my home machine. > > I believe the directory hierarchy has a lot to do with the performance. > It seems that the listing time is non-linear although I do not believe its > an N^^2 kind of problem. > I would have said the same as you before I started having to deal with 10's > of millions of files. > > > The first question I would have is how big are the actual files versus space used. Most file systems try to merge files that don't allocate in a single block to save blocks and get better space usage. Assuming a) the size of the files being rather small and b) The amount, I would be curious if ReiserFS or brtfs helps either as most have much better tail merging into metadata blocks to my knowledge. Better packing the small files can help as disk seeks are a problem here it seems. My disks will solve to like 10 MB/S on ext4 with lots of small files like this due to seeks. The other question as pointed out here is how much memory the page cache and other kernel caches are using. I would check /proc/meminfo to start as this may be another logical solution already pointed out. Maybe that helps, Nick --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On 5/4/20 1:26 PM, Lennart Sorensen via talk wrote: On Mon, May 04, 2020 at 04:38:28PM +0200, ac via talk wrote: Hi Alvin, On a 2TB dataset, with +-600k files, I have piped tree to less with limited joy, it took a few hours and at least I could search for what I was looking for... - 15TB and 100M is another animal though and as disk i/o will be your bottleneck, anything will take long, no? now, for my own info/interest, can you tell me which fs is used for this ext3? Hmm, sounds awful slow. Just for fun I ran find on one of my drives: # time find /data | wc -l 1825463 real3m57s.208s That is with 5.3T used out of 6.0TB. Running it a second time when it is cached takes 7.7s. Tree takes 14.7s. Another volume: # time find /mythdata | wc -l 54972 real0m1.924s That is with 15 TB out of 15 TB in use (yes that one always fills up for some reason). Both of those are lvm volumes with ext4 on top of software raid6 using 5400rpm WD red drives. Seems either XFS is unbelievable bad, or there isn't enough ram to cache the filesystem metadata if you are having a problem with 100M files. I only have a measly 32GB in my home machine. I believe the directory hierarchy has a lot to do with the performance. It seems that the listing time is non-linear although I do not believe its an N^^2 kind of problem. I would have said the same as you before I started having to deal with 10's of millions of files. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On 5/4/20 12:52 PM, John Sellens wrote: On Mon, 2020/05/04 12:03:19PM -0400, Alvin Starr wrote: | The client really only wants to use Centos/RHEL and ZFS is not part of that | mix at the moment. Well, one could argue that zfs on centos is fairly well supported ... If it were purely my choice I would agree but the general rule is software that can be gotten from Centos/RH and EPEL where necessary. | The data is actually sitting on a replicated Gluster cluster so trying to | replace that with an HA NAS would start to get expensive if it were a | commercial product. Of course "expensive" depends on the client. An HA truenas that size, all flash is (I believe likely well) under $15K USD. If things moved to a commercial NAS it would likely be something like Equalogic or Netapp and in that world if you have to ask the price you cannot afford it. Ah - you didn't mention Gluster. In theory, Gluster has geographic replication. It does have replication but replication is not backup. a little oops like "rm -rf * somefile" will make for a very bad day if you don't catch it before it gets replicated. And if your bricks are on LVM storage, you can use gluster snapshots as well: https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Snapshots/ to guard against accidental removals, etc. Gluster snapshots require thinly provisioned LVM. Which is doable but will require rebuilding the systems with a different LVM config. (I've not used either, and my glusters are quite old versions at the present time.) Depending on how it's all configured, you may get better performance backing up the bricks, rather than backing up gluster itself. I have a two-node gluster, mirrored, so I can backup the bricks on one of the servers and get everything. Obviously that's a very simple "cluster". Traditionally, gluster filesystem performance with large numbers of small files in a directory is horrible/pathetic. If you're backing up the gluster filesystem, you would almost certainly get better performance if your file structure is deeper and narrower, if that's possible. We are actually backing up the bricks but the they are BIG bricks. Yes gluster has a high per file open cost but in this application that is not an issue during operation. A backup using a gluster mount would move from the world of days to weeks because of the synchronization overhead. Even with snapshots the length of time for a backup can be outrageously long. Cheers John -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On Mon, May 04, 2020 at 04:38:28PM +0200, ac via talk wrote: > Hi Alvin, > > On a 2TB dataset, with +-600k files, I have piped tree to less with > limited joy, it took a few hours and at least I could search for > what I was looking for... - 15TB and 100M is another animal though > and as disk i/o will be your bottleneck, anything will take long, no? > > now, for my own info/interest, can you tell me which fs is used for this > ext3? Hmm, sounds awful slow. Just for fun I ran find on one of my drives: # time find /data | wc -l 1825463 real3m57s.208s That is with 5.3T used out of 6.0TB. Running it a second time when it is cached takes 7.7s. Tree takes 14.7s. Another volume: # time find /mythdata | wc -l 54972 real0m1.924s That is with 15 TB out of 15 TB in use (yes that one always fills up for some reason). Both of those are lvm volumes with ext4 on top of software raid6 using 5400rpm WD red drives. Seems either XFS is unbelievable bad, or there isn't enough ram to cache the filesystem metadata if you are having a problem with 100M files. I only have a measly 32GB in my home machine. -- Len Sorensen --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On Mon, 2020/05/04 12:03:19PM -0400, Alvin Starr wrote: | The client really only wants to use Centos/RHEL and ZFS is not part of that | mix at the moment. Well, one could argue that zfs on centos is fairly well supported ... | The data is actually sitting on a replicated Gluster cluster so trying to | replace that with an HA NAS would start to get expensive if it were a | commercial product. Of course "expensive" depends on the client. An HA truenas that size, all flash is (I believe likely well) under $15K USD. Ah - you didn't mention Gluster. In theory, Gluster has geographic replication. And if your bricks are on LVM storage, you can use gluster snapshots as well: https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Snapshots/ to guard against accidental removals, etc. (I've not used either, and my glusters are quite old versions at the present time.) Depending on how it's all configured, you may get better performance backing up the bricks, rather than backing up gluster itself. I have a two-node gluster, mirrored, so I can backup the bricks on one of the servers and get everything. Obviously that's a very simple "cluster". Traditionally, gluster filesystem performance with large numbers of small files in a directory is horrible/pathetic. If you're backing up the gluster filesystem, you would almost certainly get better performance if your file structure is deeper and narrower, if that's possible. Cheers John --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
On 2020-05-04 09:55, Alvin Starr via talk wrote: > > I am hoping someone has seen this kind of problem before and knows of a > solution. > I have a client who has file systems filled with lots of small files on > the orders of hundreds of millions of files. > Running something like a find on filesystem takes the better part of a > week so any kind of directory walking backup tool will take even longer > to run. > The actual data-size for 100M files is on the order of 15TB so there is > a lot of data to backup but the data only increases on the order of tens > to hundreds of MB a day. > > > Even things like xfsdump take a long time. > For example I tried xfsdump on a 50M file set and it took over 2 days to > complete. > > The only thing that seems to be workable is Veeam. > It will run an incremental volume snapshot in a few hours a night but I > dislike adding proprietary kernel modules into the systems. If you have a list of inodes on the filesystem you can use xfs_db directly: xfs_db> inode 128 xfs_db> blockget -n xfs_db> ncheck 131 dir/. 132 dir/test2/foo/. 133 dir/test2/foo/bar 65664 dir/test1/. 65665 dir/test1/foo 65666 dir/test3/foo/. 142144 dir/test2/. 142145 dir/test3/foo/bar/. 142146 dir/test3/foo/bar/baz 196736 dir/test3/. I don't know how that will perform relative to something like find though. Cheers, Jamon --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
Sadly this one is a bit of a non-starter. The client really only wants to use Centos/RHEL and ZFS is not part of that mix at the moment. The data is actually sitting on a replicated Gluster cluster so trying to replace that with an HA NAS would start to get expensive if it were a commercial product. On 5/4/20 11:27 AM, John Sellens wrote: I bet no one would want this advice, but it seems to me that the implementation needs to change i.e. that one big (possibly shallow) filesystem on xfs is unworkable. The best answer of course depends on the value of the data. One obvious approach is to use a filesystem/NAS with off-site replication. Typically a commerical product. At relatively modest cost, I like the truenas systems from ixsystems.com. ZFS based, HA versions available, replication can be done. The HA versions are two servers in one chassis, with dual-ported SAS disks. For do-it-yourselfers, I like using ZFS and ZFS replication of snapshots. For example, I do much (much) smaller offsites from my home to work using ZFS and zfs-replicate. You can also do freenas (non-commercial truenas) but without the HA hardware and code. Hope that helps - cheers John On Mon, 2020/05/04 09:55:51AM -0400, Alvin Starr via talk wrote: | The actual data-size for 100M files is on the order of 15TB so there is a | lot of data to backup but the data only increases on the order of tens to | hundreds of MB a day. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
I bet no one would want this advice, but it seems to me that the implementation needs to change i.e. that one big (possibly shallow) filesystem on xfs is unworkable. The best answer of course depends on the value of the data. One obvious approach is to use a filesystem/NAS with off-site replication. Typically a commerical product. At relatively modest cost, I like the truenas systems from ixsystems.com. ZFS based, HA versions available, replication can be done. The HA versions are two servers in one chassis, with dual-ported SAS disks. For do-it-yourselfers, I like using ZFS and ZFS replication of snapshots. For example, I do much (much) smaller offsites from my home to work using ZFS and zfs-replicate. You can also do freenas (non-commercial truenas) but without the HA hardware and code. Hope that helps - cheers John On Mon, 2020/05/04 09:55:51AM -0400, Alvin Starr via talk wrote: | The actual data-size for 100M files is on the order of 15TB so there is a | lot of data to backup but the data only increases on the order of tens to | hundreds of MB a day. --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
I am not quite sure where the breaking point is but I think part of the problem is that the directories start to get big. The directory hierarchy is only 5 to 10 nodes deep. Its running on xfs. On 5/4/20 10:38 AM, ac wrote: Hi Alvin, On a 2TB dataset, with +-600k files, I have piped tree to less with limited joy, it took a few hours and at least I could search for what I was looking for... - 15TB and 100M is another animal though and as disk i/o will be your bottleneck, anything will take long, no? now, for my own info/interest, can you tell me which fs is used for this ext3? On Mon, 4 May 2020 09:55:51 -0400 Alvin Starr via talk wrote: I am hoping someone has seen this kind of problem before and knows of a solution. I have a client who has file systems filled with lots of small files on the orders of hundreds of millions of files. Running something like a find on filesystem takes the better part of a week so any kind of directory walking backup tool will take even longer to run. The actual data-size for 100M files is on the order of 15TB so there is a lot of data to backup but the data only increases on the order of tens to hundreds of MB a day. Even things like xfsdump take a long time. For example I tried xfsdump on a 50M file set and it took over 2 days to complete. The only thing that seems to be workable is Veeam. It will run an incremental volume snapshot in a few hours a night but I dislike adding proprietary kernel modules into the systems. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] On the subject of backups.
Hi Alvin, On a 2TB dataset, with +-600k files, I have piped tree to less with limited joy, it took a few hours and at least I could search for what I was looking for... - 15TB and 100M is another animal though and as disk i/o will be your bottleneck, anything will take long, no? now, for my own info/interest, can you tell me which fs is used for this ext3? On Mon, 4 May 2020 09:55:51 -0400 Alvin Starr via talk wrote: > I am hoping someone has seen this kind of problem before and knows of > a solution. > I have a client who has file systems filled with lots of small files > on the orders of hundreds of millions of files. > Running something like a find on filesystem takes the better part of > a week so any kind of directory walking backup tool will take even > longer to run. > The actual data-size for 100M files is on the order of 15TB so there > is a lot of data to backup but the data only increases on the order > of tens to hundreds of MB a day. > > > Even things like xfsdump take a long time. > For example I tried xfsdump on a 50M file set and it took over 2 days > to complete. > > The only thing that seems to be workable is Veeam. > It will run an incremental volume snapshot in a few hours a night but > I dislike adding proprietary kernel modules into the systems. > > --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
[GTALUG] On the subject of backups.
I am hoping someone has seen this kind of problem before and knows of a solution. I have a client who has file systems filled with lots of small files on the orders of hundreds of millions of files. Running something like a find on filesystem takes the better part of a week so any kind of directory walking backup tool will take even longer to run. The actual data-size for 100M files is on the order of 15TB so there is a lot of data to backup but the data only increases on the order of tens to hundreds of MB a day. Even things like xfsdump take a long time. For example I tried xfsdump on a 50M file set and it took over 2 days to complete. The only thing that seems to be workable is Veeam. It will run an incremental volume snapshot in a few hours a night but I dislike adding proprietary kernel modules into the systems. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
