Re: Memory consumption for rsync -axv --delete
Hi, On Sun, Mar 27, 2016 at 06:39:53PM -0700, drsalists wrote: > > You could probably use CIFS, NFS or sshfs. ??It wouldn't be as fast, but the > memory requirements should be less. Sorry, I don't understand how to sync over NFS or sshfs without rsync. Can you explain this please? I don't want one copy of the data- I want a backup, but I don't want the backup to have extraneous files that I have deleted from the source. Thanks, /jl -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Memory consumption for rsync -axv --delete
You could probably use CIFS, NFS or sshfs. It wouldn't be as fast, but the memory requirements should be less. Sent from my T-Mobile 4G LTE Device Original message From: John Long <codeb...@inbox.lv> Date:2016/03/25 04:10 (GMT-08:00) To: rsync@lists.samba.org Cc: Subject: Re: Memory consumption for rsync -axv --delete On Fri, Mar 25, 2016 at 09:54:14AM +, John Long wrote: > Hi, > > I have been using rsync for many years and never had any kind of problem. > Lately I am running out of RAM trying to do an incremental backup to a box > that only has 2G of RAM. The entire directory structure I'm mirroring is > about 200G of files. A minority of subdirectories have many files. > > Is there a way to do an incremental backup with --delete option that does > not use as much memory? Is there a way to tell rsync to use a tempfile > instead of RAM for keeping tracking of whatever it does? > > And would it be useful to add ignores for the subdirectories I know have > many files and back them up separately? Is --delete safe to use in this > case, as in does --delete with --ignore somedir/ not delete files in other > target dirs that are not in the ignore path? I didn't phrase this part very well. Is --delete safe to use with --ignore, meaning will rsync avoid deleting files in the ignore path on the target side? I think the answer is probably yes but since I'm crashing the target box with --delete I don't want to have to try this too many times. Really I'm looking for a workaround to the high memory consumption so I can sync up the file trees without exceeding the small RAM capacity of the target box. Any suggestions appreciated. Thanks, /jl -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Memory consumption for rsync -axv --delete
Thank you. On Sun, Mar 27, 2016 at 02:58:49AM -0400, Kevin Korb wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > You will have an old backup dir and a new backup dir. The new one > will contain all the current stuff. The old one will contain what was > current the last time you ran rsync. Just rm -rf the old one. Or > keep a few. Or a few dozen. > > On 03/27/2016 02:54 AM, John Long wrote: > > Thanks I'll look this up. There is still the issue of how to get > > the target box cleaned up since I can no longer run --delete. > > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Memory consumption for rsync -axv --delete
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 You will have an old backup dir and a new backup dir. The new one will contain all the current stuff. The old one will contain what was current the last time you ran rsync. Just rm -rf the old one. Or keep a few. Or a few dozen. On 03/27/2016 02:54 AM, John Long wrote: > Thanks I'll look this up. There is still the issue of how to get > the target box cleaned up since I can no longer run --delete. > > /jl > > On Sun, Mar 27, 2016 at 02:49:02AM -0400, Kevin Korb wrote: You > miss-understand the purpose of --link-dest. Yes, it gives you > multiple complete backups, but each only consumes the disk space > needed to store files that are unique to that backup. Files that > are the same in 2 backup runs are actually the same file in > multiple directories requiring only 1 to actually be stored. > > On 03/27/2016 02:39 AM, John Long wrote: Hi, On Fri, Mar 25, 2016 at 11:16:47AM -0400, Kevin Korb wrote: > If you were using --link-dest to make multiple backups you > wouldn't need --delete because the target is always a new > empty directory (with - --link-dest pointing to the > previous backup run). The source is around 200G and the target box only has 500G total and some of it is used for other data. What I want to do is mirror the source on the target and be able to prune the files that get deleted from the source from the target also. I don't have enough space to back up the whole thing and that is very time consuming anyway over 100M/b link which is why I was using --delete. For a long time it was ok, but now I don't have enough RAM. There is one giant directory that is probably problematic because it has a huge number of files. I suspect this is the one that's causing me problems but it is relatively static. I suppose it could be backed up and cleaned up separately. Is there any way to reduce RAM consumption on the target box while still getting the benefit of the --delete function? I am thinking of trying to back up everything but the gigantic directory with a large number of files, and then backing up only that directory. Is this a reasonable strategy? I just couldn't understand if --delete with --exclude would delete files from the target outside the --exclude path. I guess the answer is no but it would be a very time consuming mistake. I'm trying to make sure before I try it. Thanks for your help and I'm sorry for my poorly worded post(s). /jl > >> >> -- Please use reply-all for most replies to avoid omitting the >> mailing list. To unsubscribe or change options: >> https://lists.samba.org/mailman/listinfo/rsync Before posting, >> read: http://www.catb.org/~esr/faqs/smart-questions.html > - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlb3hKgACgkQVKC1jlbQAQdqCgCdGPKy5BJGKP5oqwHFIt+I6zvw dgcAoJGkHyaai+LWPIWXiCDwTxz8V21+ =Gpd0 -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Memory consumption for rsync -axv --delete
Thanks I'll look this up. There is still the issue of how to get the target box cleaned up since I can no longer run --delete. /jl On Sun, Mar 27, 2016 at 02:49:02AM -0400, Kevin Korb wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > You miss-understand the purpose of --link-dest. Yes, it gives you > multiple complete backups, but each only consumes the disk space > needed to store files that are unique to that backup. Files that are > the same in 2 backup runs are actually the same file in multiple > directories requiring only 1 to actually be stored. > > On 03/27/2016 02:39 AM, John Long wrote: > > Hi, > > > > On Fri, Mar 25, 2016 at 11:16:47AM -0400, Kevin Korb wrote: > > > >> If you were using --link-dest to make multiple backups you > >> wouldn't need --delete because the target is always a new empty > >> directory (with - --link-dest pointing to the previous backup > >> run). > > > > The source is around 200G and the target box only has 500G total > > and some of it is used for other data. What I want to do is mirror > > the source on the target and be able to prune the files that get > > deleted from the source from the target also. I don't have enough > > space to back up the whole thing and that is very time consuming > > anyway over 100M/b link which is why I was using --delete. For a > > long time it was ok, but now I don't have enough RAM. > > > > There is one giant directory that is probably problematic because > > it has a huge number of files. I suspect this is the one that's > > causing me problems but it is relatively static. I suppose it could > > be backed up and cleaned up separately. > > > > Is there any way to reduce RAM consumption on the target box while > > still getting the benefit of the --delete function? I am thinking > > of trying to back up everything but the gigantic directory with a > > large number of files, and then backing up only that directory. Is > > this a reasonable strategy? I just couldn't understand if --delete > > with --exclude would delete files from the target outside the > > --exclude path. I guess the answer is no but it would be a very > > time consuming mistake. I'm trying to make sure before I try it. > > > > Thanks for your help and I'm sorry for my poorly worded post(s). > > > > /jl > > > > - -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > Kevin Korb Phone:(407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. ke...@futurequest.net (work) > Orlando, Floridak...@sanitarium.net (personal) > Web page: http://www.sanitarium.net/ > PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > -BEGIN PGP SIGNATURE- > Version: GnuPG v2 > > iEYEARECAAYFAlb3gl4ACgkQVKC1jlbQAQdZ7gCfXzhoaTSU2eW8fFUZQgA/8pjN > JQIAn3fQ6ESTB1gm1LT2AsO9kK4MV4ow > =Y4bU > -END PGP SIGNATURE- > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Memory consumption for rsync -axv --delete
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 You miss-understand the purpose of --link-dest. Yes, it gives you multiple complete backups, but each only consumes the disk space needed to store files that are unique to that backup. Files that are the same in 2 backup runs are actually the same file in multiple directories requiring only 1 to actually be stored. On 03/27/2016 02:39 AM, John Long wrote: > Hi, > > On Fri, Mar 25, 2016 at 11:16:47AM -0400, Kevin Korb wrote: > >> If you were using --link-dest to make multiple backups you >> wouldn't need --delete because the target is always a new empty >> directory (with - --link-dest pointing to the previous backup >> run). > > The source is around 200G and the target box only has 500G total > and some of it is used for other data. What I want to do is mirror > the source on the target and be able to prune the files that get > deleted from the source from the target also. I don't have enough > space to back up the whole thing and that is very time consuming > anyway over 100M/b link which is why I was using --delete. For a > long time it was ok, but now I don't have enough RAM. > > There is one giant directory that is probably problematic because > it has a huge number of files. I suspect this is the one that's > causing me problems but it is relatively static. I suppose it could > be backed up and cleaned up separately. > > Is there any way to reduce RAM consumption on the target box while > still getting the benefit of the --delete function? I am thinking > of trying to back up everything but the gigantic directory with a > large number of files, and then backing up only that directory. Is > this a reasonable strategy? I just couldn't understand if --delete > with --exclude would delete files from the target outside the > --exclude path. I guess the answer is no but it would be a very > time consuming mistake. I'm trying to make sure before I try it. > > Thanks for your help and I'm sorry for my poorly worded post(s). > > /jl > - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlb3gl4ACgkQVKC1jlbQAQdZ7gCfXzhoaTSU2eW8fFUZQgA/8pjN JQIAn3fQ6ESTB1gm1LT2AsO9kK4MV4ow =Y4bU -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Memory consumption for rsync -axv --delete
Hi, On Fri, Mar 25, 2016 at 11:16:47AM -0400, Kevin Korb wrote: > If you were using --link-dest to make multiple backups you wouldn't > need --delete because the target is always a new empty directory (with > - --link-dest pointing to the previous backup run). The source is around 200G and the target box only has 500G total and some of it is used for other data. What I want to do is mirror the source on the target and be able to prune the files that get deleted from the source from the target also. I don't have enough space to back up the whole thing and that is very time consuming anyway over 100M/b link which is why I was using --delete. For a long time it was ok, but now I don't have enough RAM. There is one giant directory that is probably problematic because it has a huge number of files. I suspect this is the one that's causing me problems but it is relatively static. I suppose it could be backed up and cleaned up separately. Is there any way to reduce RAM consumption on the target box while still getting the benefit of the --delete function? I am thinking of trying to back up everything but the gigantic directory with a large number of files, and then backing up only that directory. Is this a reasonable strategy? I just couldn't understand if --delete with --exclude would delete files from the target outside the --exclude path. I guess the answer is no but it would be a very time consuming mistake. I'm trying to make sure before I try it. Thanks for your help and I'm sorry for my poorly worded post(s). /jl -- ASCII ribbon campaign ( ) Powered by Lemote Fuloong against HTML e-mail X Loongson MIPS and OpenBSD and proprietary/ \http://www.mutt.org attachments / \ Code Blue or Go Home! Encrypted email preferred PGP Key 2048R/DA65BC04 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Memory consumption for rsync -axv --delete
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 If you were using --link-dest to make multiple backups you wouldn't need --delete because the target is always a new empty directory (with - --link-dest pointing to the previous backup run). So, you get the benefit of having multiple backups to restore from and rsync doesn't have to --delete. When you run low on space you just rm - -rf some old backups (takes a while but doesn't need much RAM). On 03/25/2016 05:54 AM, John Long wrote: > Hi, > > I have been using rsync for many years and never had any kind of > problem. Lately I am running out of RAM trying to do an incremental > backup to a box that only has 2G of RAM. The entire directory > structure I'm mirroring is about 200G of files. A minority of > subdirectories have many files. > > Is there a way to do an incremental backup with --delete option > that does not use as much memory? Is there a way to tell rsync to > use a tempfile instead of RAM for keeping tracking of whatever it > does? > > And would it be useful to add ignores for the subdirectories I know > have many files and back them up separately? Is --delete safe to > use in this case, as in does --delete with --ignore somedir/ not > delete files in other target dirs that are not in the ignore path? > > Thanks, > > /jl > - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlb1Vl8ACgkQVKC1jlbQAQccMQCfR+5LfpqH9to3D1QDBDScZOBX RIQAn1BB2vFbf7eDgyy7HSS2SITBTWCM =BYru -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Memory consumption for rsync -axv --delete
On Fri, Mar 25, 2016 at 09:54:14AM +, John Long wrote: > Hi, > > I have been using rsync for many years and never had any kind of problem. > Lately I am running out of RAM trying to do an incremental backup to a box > that only has 2G of RAM. The entire directory structure I'm mirroring is > about 200G of files. A minority of subdirectories have many files. > > Is there a way to do an incremental backup with --delete option that does > not use as much memory? Is there a way to tell rsync to use a tempfile > instead of RAM for keeping tracking of whatever it does? > > And would it be useful to add ignores for the subdirectories I know have > many files and back them up separately? Is --delete safe to use in this > case, as in does --delete with --ignore somedir/ not delete files in other > target dirs that are not in the ignore path? I didn't phrase this part very well. Is --delete safe to use with --ignore, meaning will rsync avoid deleting files in the ignore path on the target side? I think the answer is probably yes but since I'm crashing the target box with --delete I don't want to have to try this too many times. Really I'm looking for a workaround to the high memory consumption so I can sync up the file trees without exceeding the small RAM capacity of the target box. Any suggestions appreciated. Thanks, /jl -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Memory consumption for rsync -axv --delete
On Fri 25 Mar 2016, John Long wrote: > > I have been using rsync for many years and never had any kind of problem. > Lately I am running out of RAM trying to do an incremental backup to a box > that only has 2G of RAM. The entire directory structure I'm mirroring is > about 200G of files. A minority of subdirectories have many files. > > Is there a way to do an incremental backup with --delete option that does > not use as much memory? Is there a way to tell rsync to use a tempfile > instead of RAM for keeping tracking of whatever it does? No to the last question; you could consider adding (more) swapspace to the system, which is effectively like using a tempfile. > And would it be useful to add ignores for the subdirectories I know have > many files and back them up separately? Is --delete safe to use in this > case, as in does --delete with --ignore somedir/ not delete files in other > target dirs that are not in the ignore path? There's no --ignore, you probably mean --exclude. I don't really understand what you're asking in your last question... Why should --exclude somedir/ affect what --delete does elsewhere? --delete will still delete stuff elsewhere if necessary. Also look at the description of --delete and --delete-excluded, if you have any questions about what's in the manpage then feel free to ask those here; but for now I get the impression you haven't spent much time reading the manpage. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Memory consumption for rsync -axv --delete
Hi, I have been using rsync for many years and never had any kind of problem. Lately I am running out of RAM trying to do an incremental backup to a box that only has 2G of RAM. The entire directory structure I'm mirroring is about 200G of files. A minority of subdirectories have many files. Is there a way to do an incremental backup with --delete option that does not use as much memory? Is there a way to tell rsync to use a tempfile instead of RAM for keeping tracking of whatever it does? And would it be useful to add ignores for the subdirectories I know have many files and back them up separately? Is --delete safe to use in this case, as in does --delete with --ignore somedir/ not delete files in other target dirs that are not in the ignore path? Thanks, /jl -- ASCII ribbon campaign ( ) Powered by Lemote Fuloong against HTML e-mail X Loongson MIPS and OpenBSD and proprietary/ \http://www.mutt.org attachments / \ Code Blue or Go Home! Encrypted email preferred PGP Key 2048R/DA65BC04 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
