Re: incremental FTP backup program
On Thu, 17 Dec 2009 17:29:54 +0100 bsd b...@todoo.biz wrote: Hello, I have been provided with a 1To backup space by my hosting company… unfortunately It is only accessible by FTP which does not allow me to do the regular backup / restore I use to do using RSYNC. What are the options I have in order to do the backup / restore of the files knowing that : 1. I would like to have a time based backup solution (incremental with sub-folder for daily backup so I can restore files from xxx days ago). 2. I need to use ftp as a transport 3. Some of my servers have limited space so doing a local sync then FTP will not be possible. 4. If possible I would like a bullet proof solution ideally well known | tested and available from the port tree. In fact what I would be looking for is something like rsnapshot but with FTP transport / compatibility. Have a look at duplicity (http://duplicity.nongnu.org/). It can use ftp as transport and uses the rsync algorithm so only the changed parts of files are sent to the archive when doing an incremental backup. Manfred ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: incremental FTP backup program
Thanks Manfred ! This program really rocks. Backup up are fast and efficient… Learning curve is not so steep, seems very good to me. Encrypts backup with GPG; looks perfect. Thx everyone for your answers. Le 18 déc. 2009 à 10:16, Manfred Usselmann a écrit : On Thu, 17 Dec 2009 17:29:54 +0100 bsd b...@todoo.biz wrote: Hello, I have been provided with a 1To backup space by my hosting company… unfortunately It is only accessible by FTP which does not allow me to do the regular backup / restore I use to do using RSYNC. What are the options I have in order to do the backup / restore of the files knowing that : 1. I would like to have a time based backup solution (incremental with sub-folder for daily backup so I can restore files from xxx days ago). 2. I need to use ftp as a transport 3. Some of my servers have limited space so doing a local sync then FTP will not be possible. 4. If possible I would like a bullet proof solution ideally well known | tested and available from the port tree. In fact what I would be looking for is something like rsnapshot but with FTP transport / compatibility. Have a look at duplicity (http://duplicity.nongnu.org/). It can use ftp as transport and uses the rsync algorithm so only the changed parts of files are sent to the archive when doing an incremental backup. Manfred ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org Gregober --- PGP ID -- 0x1BA3C2FD bsd @at@ todoo.biz P Please consider your environmental responsibility before printing this e-mail ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: incremental FTP backup program
On Thu, Dec 17, 2009 at 05:29:54PM +0100, bsd typed: Hello, I have been provided with a 1To backup space by my hosting company? unfortunately It is only accessible by FTP which does not allow me to do the regular backup / restore I use to do using RSYNC. What are the options I have in order to do the backup / restore of the files knowing that : 1. I would like to have a time based backup solution (incremental with sub-folder for daily backup so I can restore files from xxx days ago). 2. I need to use ftp as a transport 3. Some of my servers have limited space so doing a local sync then FTP will not be possible. 4. If possible I would like a bullet proof solution ideally well known | tested and available from the port tree. In fact what I would be looking for is something like rsnapshot but with FTP transport / compatibility. Never tried, bet maybe sysutils/fusefs-curlftpfs will work for you. Ruben ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: incremental FTP backup program
On Thu, Dec 17, 2009 at 05:29:54PM +0100, bsd wrote: Hello, I have been provided with a 1To backup space by my hosting company… unfortunately It is only accessible by FTP which does not allow me to do the regular backup / restore I use to do using RSYNC. What are the options I have in order to do the backup / restore of the files knowing that : 1. I would like to have a time based backup solution (incremental with sub-folder for daily backup so I can restore files from xxx days ago). 2. I need to use ftp as a transport 3. Some of my servers have limited space so doing a local sync then FTP will not be possible. 4. If possible I would like a bullet proof solution ideally well known | tested and available from the port tree. There are several avenues available even with the tools in the base system. One of those is to use dump(8), assuming you are using UFS filesystems, of course. Make regular dumps (of different levels), compress them using bzip2(1), encrypt the compressed dumpfiles using openssl(1) (do _not_ use crypt(1)!) and use ftp(1) to transfer them to the hosting company. Another way is to use mtree(8). You can use this to generate a file containing the state of a filesystem hierarchy. Running it against a previously saved specification can yield a list of files that have changed. Use tar and bzip2 to create an archive of the files that have changed. Encrypt and send same as above. All these programs come with the base system, you would just need to script them. Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) pgpJa2eZrTxP0.pgp Description: PGP signature
Re: incremental FTP backup program
Are there any specific scripting tools that I could use in order to achieve that ? From what you are describing a tool that would automate the dump process and take care of snapshots versions would be a must… ?? Any idea ? Le 17 déc. 2009 à 18:12, Roland Smith a écrit : On Thu, Dec 17, 2009 at 05:29:54PM +0100, bsd wrote: Hello, I have been provided with a 1To backup space by my hosting company… unfortunately It is only accessible by FTP which does not allow me to do the regular backup / restore I use to do using RSYNC. What are the options I have in order to do the backup / restore of the files knowing that : 1. I would like to have a time based backup solution (incremental with sub-folder for daily backup so I can restore files from xxx days ago). 2. I need to use ftp as a transport 3. Some of my servers have limited space so doing a local sync then FTP will not be possible. 4. If possible I would like a bullet proof solution ideally well known | tested and available from the port tree. There are several avenues available even with the tools in the base system. One of those is to use dump(8), assuming you are using UFS filesystems, of course. Make regular dumps (of different levels), compress them using bzip2(1), encrypt the compressed dumpfiles using openssl(1) (do _not_ use crypt(1)!) and use ftp(1) to transfer them to the hosting company. Another way is to use mtree(8). You can use this to generate a file containing the state of a filesystem hierarchy. Running it against a previously saved specification can yield a list of files that have changed. Use tar and bzip2 to create an archive of the files that have changed. Encrypt and send same as above. All these programs come with the base system, you would just need to script them. Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) Gregober --- PGP ID -- 0x1BA3C2FD bsd @at@ todoo.biz P Please consider your environmental responsibility before printing this e-mail ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: incremental FTP backup program
On Thu, Dec 17, 2009 at 10:55:54PM +0100, bsd wrote: Are there any specific scripting tools that I could use in order to achieve that ? Well, /bin/sh and 'man sh' spring to mind. :-) Or use another scripting language if you are more familiar with that, e.g. perl or python. But since you are mainly invoking programs, a plain shell-script would be my first choice. From what you are describing a tool that would automate the dump process and take care of snapshots versions would be a must… ?? See e.g. the dodumps script on my scripts page; http://www.xs4all.nl/~rsmith/software/scripts.html It just names the dumps 'filesystem-dumplevel-date.dump', e.g. root-0-20091217.dump. You could extend this script to transfer the dumps via ftp, and remove old dumps from the FTP site. After testing, you could even run it from cron. Start off with a level 0 dump, and then perform higher level dumps as often as you need. Google for 'dump levels', and you'll find lots of different schemes, e.g. http://docs.sun.com/app/docs/doc/817-5093/bkupconcepts-21?a=view I would urge you to keep things simple. The more complicated the your solution, the easier things can go wrong. One caveat about dump though. If one of your filesystems contains just one big database in a huge file, dump will copy the whole file even if just one byte has changed. In such a case you should see if the database has tools to just copy the records changed since the last backup and use that instead. Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) pgpwsaSQLwSSv.pgp Description: PGP signature
Re: incremental FTP backup program
I have a backup system that I created for rougly that scenario. It's not bullet proof, not well tested in the community, etc. I haven't even tried the FTP Transport on FreeBSD yet. http://lostlogicx.com/backupsblow/ I use it over S3 to backup my server personally. --Brandon On 2009-12-17 (Thu) at 23:39:40 +0100, Roland Smith wrote: On Thu, Dec 17, 2009 at 10:55:54PM +0100, bsd wrote: Are there any specific scripting tools that I could use in order to achieve that ? Well, /bin/sh and 'man sh' spring to mind. :-) Or use another scripting language if you are more familiar with that, e.g. perl or python. But since you are mainly invoking programs, a plain shell-script would be my first choice. From what you are describing a tool that would automate the dump process and take care of snapshots versions would be a must??? ?? See e.g. the dodumps script on my scripts page; http://www.xs4all.nl/~rsmith/software/scripts.html It just names the dumps 'filesystem-dumplevel-date.dump', e.g. root-0-20091217.dump. You could extend this script to transfer the dumps via ftp, and remove old dumps from the FTP site. After testing, you could even run it from cron. Start off with a level 0 dump, and then perform higher level dumps as often as you need. Google for 'dump levels', and you'll find lots of different schemes, e.g. http://docs.sun.com/app/docs/doc/817-5093/bkupconcepts-21?a=view I would urge you to keep things simple. The more complicated the your solution, the easier things can go wrong. One caveat about dump though. If one of your filesystems contains just one big database in a huge file, dump will copy the whole file even if just one byte has changed. In such a case you should see if the database has tools to just copy the records changed since the last backup and use that instead. Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
