Re: Bug when using rsync -r --delete -b --backup-dir

[Henri Shustak via rsync]

You may wish to take a look at LBackup : http://www.lbackup.orgPlease note as one of the developers I am biased.On 24/11/2020, at 10:33 AM, CRTS via rsync  wrote:Consider following directory structure:mkdir -p src/empty tgt bkpwhen making an initial backup withrsync -r src/ tgtthen the empty directory in src will be copied to tgt. However, if thedirectory is removed from srcrmdir src/emptyand rsync is run likersync -r --delete -b --backup-dir=../bkp src/ tgt/then the empty directory does end up in bkp. I am using the --backup-dirand the --delete option to move an outdated snapshot to another directoryby backing up an empty directory. I was expecting that the entiredirectory structure to be preserved. Apparently this is not the casefor empty directories.Can this be fixed? Is this even the right place to file a bug report? Ido not have a github account to do this on github.-- Please use reply-all for most replies to avoid omitting the mailing list.To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsyncBefore posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: RAM speedup

[Henri Shustak via rsync]

You may need a lot of ram. Take a look at how much RAM ZFS needs to operate 
well.

Henri

> On 27/06/2020, at 7:58 AM, Rupert Gallagher via rsync  
> wrote:
> 
> Hello, 
> 
> As disks are slow and rsync reads and writes so much that for the bus this is 
> the equivalent of context switching galore, would it be possible to use RAM 
> as a buffer? Say, you have 10GB of spare RAM, rsync uses the bus to its peak 
> for reading 10GB, then again for writing it down. This would be more 
> efficient than lot of small read/write operations. 
> 
> Thank you
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: RAM speedup

[Henri Shustak via rsync]

Not sure if it is useful. Will put a plug in for a rsync hard based backup 
system primarily focused on macOS. If it is speed you are after, then this is 
probably not the right / helpful tool. But feel free to pick out what ever is 
useful for you : http://www.lbackup.org

Henri

> On 29/06/2020, at 7:29 AM, Matthias Schniedermeyer via rsync 
>  wrote:
> 
> On 28.06.2020 16:46, Rupert Gallagher wrote:
>> ? Original Message ?
>> On Sunday 28 June 2020 13:58, Matthias Schniedermeyer  wrote:
>> 
>> destination:
>> ST5000LM000-2AN1 sata hdd
>> Writing speed  : 74 MB/s
>> Reading speed  : 89 MB/s
> 
> And this HDD is a SMR model(*) on top of beeing a 4k sector model emulating 
> 512 byte sectors.
> So alignment needs to be correct and the filesystems must use 4k sectors.
> 
> This HDD is NOT suitable for beeing used for small files and a hardlink-farm.
> 
> SMR HDDs only reach best performance if used "like a tape drive" with large 
> and linear writing.
> 
> I use SMR HDDs myself and only use them for "really large" files (>500MB per 
> file), otherwise they perform very poorly.
> I also align my partitions correctly and set XFS to "4k sector"-size.
> Which nowadaya means: I use 4k sectors for anything. As i only have been 
> using 4k sector HDDs for nearly as long as they are on the market (IIRC >1 
> decade).
> Also SSDs are usually optimized for 4k sectors too.
> 
> 
> *:
> https://en.wikipedia.org/wiki/Shingled_magnetic_recording
> 
> 
> -- 
> 
> Matthias
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: osx permission issue

[Henri Shustak via rsync]

If you are talking about SIP, then at present SSH will have full disk access 
(as much as it can) by default.

Take a look at this LBackup page which discusses this in more detail : 
http://www.lbackup.org/developer/dealing_with_sip

Hope that helps.



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Advanced rsync includes and excludes

[Henri Shustak via rsync]

Hello,

http://www.lbackup.org/developer/advanced_excludes_file_examples

The link above may outline a slightly different approach which may work on your 
system. However, you would need that approach assumes the backup path is set to 
root which may not be what you want to do.

For quick inline reference the excludes file example is included below. You 
should be able to modify this to make it work for you. However, as mentioned 
you will need to test it out on your system and see if it is doing what you 
expect.

> /private/etc/ssh_config/
> + private 
> + private/etc/***
> + private/var/***
> + home/***
> - *


Henri



> I'm a happy camper @ rsync (and rsnapshot) since years. Thanks for this major 
> piece of software.
> 
> In an attempt to reorganize my rsnapshot backups, I stumbled across an issue, 
> that I'm trying to seeking a more sophisticated solution here.
> 
> Given, I have a deeply branched tree, where I would like to include a 
> specific 
> directory deep under, while excluding anything else on that path, I find 
> myself doing:
> 
> + /some/
> + /some/very/
> + /some/very/deep/
> + /some/very/deep/path/
> + /some/very/deep/path/to/
> + /some/very/deep/path/to/save/
> - /some/very/deep/path/to/*
> - /some/very/deep/path/*
> - /some/very/deep/*
> - /some/very/*
> - /some/*
> 
> While it works, it feels rather awkward and gets very complicated, if you 
> have 
> more of such items to deal with. Sure, I could run this separately, but this 
> isn't the real McCoy either with a complicated tree structure.
> 
> What are your favorite ways of doing such tasks with rsync?
> 
> Thanks in advance,
> Pete
> 
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync many processes and slow backup

[Henri Shustak via rsync]

I think others have answered your questions. You may want to implement some 
sort of locking system in the script so that you can only run a single instance 
of your script at. If you are looking for a backup tool which leverages rsync 
you may want to take a look at LBackup : http://www.lbackup.org

There is even a pre-script you can enable (in the example resources) so that 
the script can run a maximum of once a day / once a month etc (even if you 
start it many times from cron.

Hope you have it all working now :)

Henri


>> Hi all! :-)
>> 
>> I have a small rsync script to sync my data to a usb-disk. It works
>> fine, when I start it in console. I get 3 rsync processes (look in
>> top) and the backup takes ~25 min. But, when I add the script in
>> cron to start it at 1am at night it takes 7 - 9 hours and I see up to
>> 180 processes. When I look in top I see a hight load of 60 - 80 and 40
>> - 60 waits. Why? Can someone explain why it takes so long when it
>> starts with cron?
>> 
>> This is my rsync command:  rsync -azc --delete "$QUELLORDNER"
>> "$ZIELORDNER" 
>> This is the entry in cron (crontab -e): * 2 * * *
>> /root/backupscript/backup.sh
>> Data to sync: 18 Gb, 185.000 files.
>> 
>> When I look in the log files I see errors like this: 
>> 
>> rsync:
>> rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c"
>> -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) 
>> 
>> rsync error: some files/attrs were not transferred (see previousrsync
>> error:
>> 
>> some files/attrs were not transferred (see previous errors) (code 23) at
>> 
>> main.c(1196) [sender=3.1.2]
>> 
>> directory (2)
>> 
>> 
>> 
>> Can you help me to solve this problem?
>> 
>> 
>> 
>> regards,
>> 
>> Hannes Hutmacher
>> 
>> 
> A couple of questions: (I am not an expert user.)
> 
> Does your manual job run as the same user (presumably root) as does your
> cron job?
> 
> Are you backing up any temporary files that you might be better off
> excluding?
> 
> Is anything else running at the same time as your cron job which may be
> creating and destroying files in the backup source or target? (E.g. Your
> firewall?)
> 
> I'm not very familiar with fuzzy searching, but using that in places
> like /var seems odd.
> 
> One of the constant refrains on this list is that using checksums is
> almost never a good idea. In combination with fuzzy, it seems even more
> tenuous.
> 
> Joe
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Huge directory tree: Get files to sync via tools like sysdig

[Henri Shustak]

That sounds like it certinally would not hurt!


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Huge directory tree: Get files to sync via tools like sysdig

[Henri Shustak]

As Ben mentioned, ZFS snapshots is one possible approach. Another approach is 
to have a faster storage system. I have seen considerable speed improvements 
with rsync on similar data sets by say upgrading the storage sub system.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: downloading only specific directories from directory tree

[Henri Shustak]

The following URL may also be helpful in terms of another example : 
http://www.lbackup.org/developer/advanced_excludes_file_examples

Hope that link is helpful.


HTRAX 2013 Revitalised : EGYPTIAN HUMP HTRAX : Direct URL download :
http://henri.shustak.org/download/htrax/egyptian-hump-htrax.mp3
"Dr Who Meets B52's" - More reviews : http://www.jessetaylor.com.au/htrax



On 21/01/2017, at 12:15 PM, Kevin Korb  wrote:

> An include only overrides an exclude that follows it.  So, you would
> need something like this:
> 
> + */
> + openSUSE_13.2/***
> - *
> 
> Then add --prune-empty-dirs to your rsync command line.  I left off the
> initial ** because it doesn't really mean anything unless openSUSE_13.2
> isn't always the entire directory name in which case 1 * would do.
> 
> Another way to explain it is that by default everything is included.  An
> exclude rule makes exceptions.  An include rule makes exceptions to the
> exclude rule.
> 
> On 01/20/2017 06:08 PM, Istvan Gabor wrote:
>> Hello:
>> 
>> I have read rsync manual and several howtos on how to use rsync, still
>> I don't know if it's doable what I want to do, and if yes, how.
>> 
>> The scenario:
>> 
>> I would like to make a local copy of openSUSE 13.2 repositories.
>> I use openSUSE linux.
>> 
>> The repos are located in a multi-level directory structure, eg:
>> 
>> ftp://ftp.halifax.rwth-aachen.de/opensuse/repositories/
>> 
>> Let's call this directory the root dir.
>> 
>> Each directory in the root has subdirectories, and one of the
>> subdirectories
>> is named "openSUSE_13.2". The openSUSE_13.2 dir can be one, two, or more
>> level
>> downstairs in the root dir, eg:
>> 
>> http://download.opensuse.org/repositories/Apache/openSUSE_13.2/
>> http://download.opensuse.org/repositories/Apache:/MirrorBrain/openSUSE_13.2/
>> 
>> 
>> There are plenty of directories in the root dir, and each subdirectory can
>> have several subdirectories.
>> 
>> Is it possible, using rsync, to download only all "openSUSE_13.2"
>> directories with
>> their contents, without specifying directly one by one every subdirectory
>> to be included or excluded?
>> 
>> I tried to use filters.txt file with content:
>> 
>> + **openSUSE_13.2/***
>> 
>> with this command:
>> 
>> rsync -av --include-from=filters.txt
>> rsync://ftp.halifax.rwth-aachen.de/opensuse/repositories/ ./
>> 
>> This downloads every directory. How can I exclude those are not
>> openSUSE_13.2?
>> 
>> Thanks,
>> 
>> Istvan
>> 
> 
> -- 
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>   Kevin Korb  Phone:(407) 252-6853
>   Systems Administrator   Internet:
>   FutureQuest, Inc.   ke...@futurequest.net  (work)
>   Orlando, Floridak...@sanitarium.net (personal)
>   Web page:   http://www.sanitarium.net/
>   PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Robustness: sometimes write times after having renamed the temp file

[Henri Shustak]

I believe some work on this was happening at one point in rsync (not specific 
to OS X).

In terms of your issues specifically, would the fuzzy option possibly help you?

> --fuzzy find similar file for basis if no dest file


Could you report if it makes a difference for your use case?

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync: connection unexpectedly closed

[Henri Shustak]

>> Have you tried performing a copy to a known good local device?  If a
>> local copy fails, then I would start checking the file system of the
>> source and also the hardware of that system.
> 
> That's a good idea. I just tried that and it copied no problem.

Do you have another system you could try this transfer with via SSH with on the 
local network?

Given that the local transfer works fine, I would suggest checking the hardware 
and file system integrity on the remote machine. In terms of hardware checking 
the memory and disks would be a top priority.

Also, you could try moving the partial file out of the way and also not using 
the partial option and transferring again?

Hope that helps


HTRAX 2013 Revitalised : EGYPTIAN HUMP HTRAX : Direct URL download :
http://henri.shustak.org/download/htrax/egyptian-hump-htrax.mp3
"Dr Who Meets B52's" - More reviews : http://www.jessetaylor.com.au




-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync failure on corrupted source data

[Henri Shustak]

+1 for ddrescue :)

> https://www.gnu.org/software/ddrescue/



This email is protected by LBackup, an open source backup solution
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync with overlay tree

[Henri Shustak]

If you are looking at overlays specifically have you checked out RADMIND?

http://rsug.itd.umich.edu/software/radmind/macosx.html

Note : The link above is more specific to the MacOSX release but 
   it is a good starting point

Hope that helps!


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

On 1/04/2016, at 3:57 PM, Perry Hutchison  wrote:

> tomr  wrote:
>> Almost all of the config of someapp has remained unchanged
>> in the latest major release.
> 
> Distributing config files really sounds more like a job for
> sysutils/puppet than for rsync :)  Teach facter to return the
> version of someapp that's installed and you don't even have
> to keep track of which machines need to have the overlay tree
> included in their rsync invocations.
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: OS X and rsync: unpack_smb_acl errors and --numeric-ids

[Henri Shustak]

> Submitted as bug:
> https://bugzilla.samba.org/show_bug.cgi?id=10649

This bug has been stale for a while. In order to dive in and have a go at 
working out what is wrong and how to fix it ; is signing up with the samba bug 
tracker required?

Probably will not have time to look at it immediately. Just (A) pinging the bug 
and (B) wondering how patches are submitted or if there is a patch already in 
existance?

Thanks
Henri



InstallPKG, an open source installation system. Install multiple apple .pkg and 
.mpkg files via the command line : http://www.lucid.technology/tools/installpkg
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Empty directories aren't backed up in backup-dir

[Henri Shustak]

Using the link-dest option would be one way to solve this.

If you are seeking a backup system which uses hard links (link-dest), then you 
may wish to take a look at LBackup 

Hope this helps.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

On 18/11/2015, at 11:46 PM, Adam Błaszczykowski  
wrote:

> Hello,
> We have strange issue with rsync and backup-dir option. In case of having 
> empty directories and deleting them, those directories are not backed up in 
> backup-dir directory. Directories are backed up only if contains files. 
> 
> Scenario:
> # mkdir -p source/b
> # rsync -a --backup --backup-dir=../backup source/ destination/
> # rm -rf source/b
> # rsync -a --backup --backup-dir=../backup source/ destination/
> # ls -la backup
> 
> Backup-dir is empty. I expected to have ‘b’ directory inside.
> How can we force rsync to backup empty directories to backup-dir in case of 
> deletion?
> 
> 
> Best Regards
> Adam Blaszczykowski
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Henri Shustak]

Just chiming in slightly off topic.

As a first step if you are going to be backing up files to some media with a 
computer it would be a really good idea to ensure, that the hardware being used 
is not faulty. I am not saying that your hardware is faulty. However, it would 
be worth checking this somehow. Check the drive media for bad blocks, check 
that all the cables are working well. Ensure the mother board of the system is 
in good working order etc.

As a second step if you are going to be performing backups (with a file system 
based tool such as rsync) to any kind of file system in future, I would 
strongly suggest checking the file system is in a good state on a regular 
basis. File system corruption is capable of cause all sorts of problems for 
backup systems which rely upon the file system like rsync.

Hope this helps.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

On 2/10/2015, at 9:48 AM, Ronald F. Guilmette  wrote:

> 
> In message <560ce706@sanitarium.net>, 
> Kevin Korb  wrote:
> 
>> Yes, when it comes to local copies cp is significantly faster than
>> rsync.  Without --link-dest there isn't much advantage to using rsync
>> for backups.  The only thing you get beyond cp -au is --delete.
> 
> I just now remembered the (forehead slap) bloody obvious reason I decided
> to use rsync to make and maintain my backup drive(s).
> 
> Yes, it theory I could have used something simpler... cp -R or else
> maybe cpio -p... but those just copy everything blindly.  For my
> backups, I only need/want to have the NEW and/or MODIFIED files
> copied to the backup drive.  (And also, of course, I need to have
> files that have been deleted on the main drive be deleted also on
> the backup drive.)
> 
> Rsync does everything I want as far as making and maintaining backups.
> I could also have used FreeBSD backup & restore programs, but for
> reasons I can't really remember anymore, I concluded that rsync was
> the better option.
> 
> 
> Regards,
> rfg
> 
> 
> P.S.  I have no idea what the -u option for cp is supposed to do.
> I guess that must be a Linux-ism.  The FreeBSD man page for cp doesn't
> mention any such thing as a -u option.
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Recycling directories and backup performance. Was: Re: rsync --link-dest won't link even if existing file is out of date (fwd)

[Henri Shustak]

 Our backup procudures have provision for looking back at previous 
 directories, but there is not much to be gained with recycled directories.  
 Without recycling, and after a failure, the latest available backup may not 
 have much in it.

Just wanted to point out that LBackup has a number of checks in place to detect 
failures during a backup. If this happens, then that backup is not labeled as a 
successful snapshot. 

At present, when the next snap shot is started, the previous incomplete 
snapshot(s) are not used as a link-dest source. As mentioned, this is something 
I have been looking at for a while. However, there are some edge cases which 
need to be handled carefully if you use incomplete backups as a link-dest 
source. I am sure these problems are all contractable, I have simply not spend 
enough time.

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org




-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Recycling directories and backup performance. Was: Re: rsync --link-dest won't link even if existing file is out of date (fwd)

[Henri Shustak]

 How do you handle snapshotting? or do you leave that to the block/fs 
 virtualization layer?


I am guessing this question is directed at me. 

Firstly, when I used the word snapshot, I was referring to the snapshot in the 
LBackup context. It is outlined at the following page 
http://www.lbackup.org/hard-links. As such, it is not a file system snapshot 
(unless you use the scripting subsystem) it is instead a backup made at a 
specific date and time.

Secondly, if you are using the scripting subsystem to take an actual snapshot 
and then possibly mount this and then use LBackup to take a snapshot of that 
then this uses the file system virtualization layer.

Apologies for the confusion using this word has caused. It is simply a snapshot 
as described and referenced though out the LBackup documentation.

Let me know if further clarification is required.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --link-dest won't link even if existing file is out of date

[Henri Shustak]

 Ill take a look but I imagine I cant backup the 80 Million files I need
 to in under the 5 hours i have for nightly maintenance/backups. Currently
 it's possible by recycling directories...

To cover that many files in that much time you will require a high speed system.
Just another thought. Perhaps splitting the backup onto multiple backup servers 
/ storage systems would reduce the backup time so that it fits into your window?

Also, I strongly agree with the previous posts relating to file system 
snapshots. ZFS is just one file system which supports this kind of system.

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --link-dest won't link even if existing file is out of date

[Henri Shustak]

Hi Ken,

You may wish to take a quick look at LBackup (disclaimer I am a developer on 
this project) which is a wrapper to rsync ; designed for reliable user data 
backups. 

LBackup always starts a new backup snapshot with an empty directory. I have 
been looking at extending --link-dest options to scan beyond just the previous 
successful backup to (failed backups / older backups). However, there are all 
kinds of edge cases which are worth considering with such a changes. At present 
LBackup is focused on reliability as such, this RD is quite slow given limited 
resources. The current version of LBackup offers IMHO reliable backups of user 
data and the scripting sub-system offers a high degree of flxibility.

Yes, every time you start a backup snapshot, a directory is re-populated from 
scratch and this takes time with LBackup. However, if you are seeking 
reliability then you may wish to check out the following URL : 
http://www.lbackup.org

If you are looking to speed up performance, then investing in faster hardware, 
additional file system caching or considering various file systems is well 
worth while.

Ideas and patches are welcome to improve the LBackup project.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync to multiple destinations

[Henri Shustak]

 I am looking for a way to start one rsync command with multiple destinations.
 My use case:
 I have one (slow) usb drive which should be synced to two other
 harddrives. It would be great if rsync could read one file which is to
 be copied to both hard drives only once, so the slow usb connection is
 used more efficiently.


Just a thought, you could rsync to faster local storage and then distribute to 
the other locations from there. This would result in a single slow read from 
the slow USB drive. 

Another thought, perhaps you could replace the slow USB drive altogether with a 
low cost upgrade?


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync with 8bit file names?

[Henri Shustak]

 When trying to sync my TV folder to a mirror drive, episodes with non-ASCII 
 characters in them cannot be processed by rsync. Anything I can do about 
 this?
 
 (Q1) What do you mean by a mirror drive? Is this a RAID1 external enclosure 
 or some sort of softRAID? Or is it just a copy of the TV folder onto another 
 device?
 
 A copy of the TV folder.

Okay, thanks for confirmation on this point. Have you confirmed that the file 
system is good on the destination directory. IE : Running Disk Utility to check 
the file system?


 (Q2) Is your Drobo's firmware all up-to date?
 Yes
Great.

 (Q3) Have you tried via SMB rather than AFP?
 No, but my understanding is that afp in 10.9 and 10.10 *si* smb.
I do not understand your reply? What is *si*?

 (Q2-1) Have you tried this same setup when booted from another OS (e.g. 
 10.9)?
 No, I moved my machine to the DP with DP7 at the same time that I added this 
 new external store to mirror my TV folder to.
I would strongly suggest trying this out on a system running 10.9 (not beta 
software) and seeing if this resolves the problem.

 (Q2-2) Have you tried this using some other hardware?
 Limited options. I suppose i could move it to a Mac mini that is still 
 running 10.9 to test, but then again, it’s about 15 files I can manually deal 
 with out of 7TB which seems like less trouble that moving hardware around.
Sure, but you have asked for assistance from the rsync mailing list. It would 
be good to have answers to the following : 
 Is there a problem on 10.10?
 Is there a problem with some of the hardware you are using?

 (Q2-3) Have you tried coping this specific file over using the Finder (say 
 to the Desktop) and then using rsync to copy it over to that destination 
 drive?
 I copied the file in the finder to the destination, I did not try copying it 
 and the using rsync locally, but I can do that.

I would give that a try and see if it makes any difference. If that resolves 
the problem then I suspect it is either something to do with the Drobo, 10.10 
or even the network?

Hope this helps.

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync with 8bit file names?

[Henri Shustak]

 When trying to sync my TV folder to a mirror drive, episodes with non-ASCII 
 characters in them cannot be processed by rsync. Anything I can do about this?

(Q1) What do you mean by a mirror drive? Is this a RAID1 external enclosure or 
some sort of softRAID? Or is it just a copy of the TV folder onto another 
device?

(Q2) Is your Drobo's firmware all up-to date?

(Q3) Have you tried via SMB rather than AFP?



This email is protected by LBackup, an open source backup solution
http://www.lbackup.org
@kreme.com wrote:

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync with 8bit file names?

[Henri Shustak]

Hello again,

Three further questions :

(Q2-1) Have you tried this same setup when booted from another OS (e.g. 10.9)?

(Q2-2) Have you tried this using some other hardware?

(Q2-3) Have you tried coping this specific file over using the Finder (say to 
the Desktop) and then using rsync to copy it over to that destination drive?


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync with 8bit file names?

[Henri Shustak]

Hello,

Guessing the source drive is formatted HFS+ or HFS+J. What is the file system 
on the destination drive? I know that rsync is capable of correctly coping over 
file and folder names on Mac OS X which contain non-ASCII characters. Hence my 
question relating to the destination file system?

The following command should provide you with an answer : 
diskutil info /Volumes/TV | grep File System Personality | awk -F File 
System Personality:   '{print $2}'

Maybe someone has a better way to work out the FileSystem volume which they can 
share? Typically there are many ways to skin a cat and that some approaches are 
more elegant than others.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

On 12/09/2014, at 12:36 AM, LuKreme krem...@kreme.com wrote:

 When trying to sync my TV folder to a mirror drive, episodes with non-ASCII 
 characters in them cannot be processed by rsync. Anything I can do about this?
 
 For example, I get this:
 
 building file list ... 
 file has vanished: /Volumes/TV/Raising Hope/Raising Hope - S04E01 - 
 De?\#201ja Vu Man.mkv
 
 This is rsync under OS X:
 
 $ rsync --version
 rsync  version 2.6.9  protocol version 29
 Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
 http://rsync.samba.org/
 Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
  inplace, IPv6, 64-bit system inums, 64-bit internal inums
 
 I'd check macports, but since it's not been updated for 10.10 yet I'll need 
 to wait a while.
 
 -- 
 Passion is the pill you can swallow forever Taking them one by one One
 by One --Agents of Good Roots Come On
 
 -- 
 Please use reply-all for most replies to avoid omitting the mailing list.
 To unsubscribe or change options: 
 https://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync with 8bit file names?

[Henri Shustak]

Hello,

Is the source Drobo a NAS or DAS unit? 

Also, I am guessing that the /Volumes/TV is the source for the rsync. Is it 
possible that the file has actually moved or been renamed?

If the Drobo (hosted) volume is the source for the rsync command, then perhaps 
checking the file system format of that device is also worth while?

Hope this helps. Perhaps someone else on the list has some ideas? 

I would suggest you consider compiling a recent version of rsync for your 
system. This should be quite straight forward on 10.10 (beta release). However, 
I must admit I have yet to try compiling rsync on 10.10 : 
http://www.lbackup.org/developer/rsync_hfs

Finally, what is the actual command you are running. This information may shed 
some light on why the command is failing.

Hope this helps.


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org

On 12/09/2014, at 12:40 PM, LuKreme krem...@kreme.com wrote:

 On 11 Sep 2014, at 16:22 , Henri Shustak henri.shus...@gmail.com wrote:
 Guessing the source drive is formatted HFS+ or HFS+J.
 
 The source drive is a Drobo. The destination is Journaled HFS+.
 
 -- 
 'I cannot! He has been kindness itself to me!' 'And you can be Death
 itself to him.'
 

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Backup scripts

[Henri Shustak]

Hello,

The --link-dest approach is the way LBackup works and this is also what allows 
the lcd command (part of LBackup) to move you back and forth in time while you 
remain in the same directory relative directory. This makes finding a 
particular data quite easy during a partial restore.

In particular, having a full directory makes restore from a point in time easy 
and (as Kevin pointed out) there is no need to merge various incremental 
backups. Each point it time simply appears as a full file system tree. 
Although, the space required is lowered somewhat due to the use of hard links.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

On 5/09/2014, at 9:44 AM, Kevin Korb k...@sanitarium.net wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 That is how to do an incremental backup.  But rsync can do a snapshot
 with --link-dest instead of --backup.  The big difference is that with
 - --link-dest each backup will contain the entire tree not just the
 different files.  Therefore if you need to restore an old version you
 can do so without merging the current with the old.
 
 Also, no -z on a local copy ;)
 
 On 09/04/2014 05:41 PM, Greg Deback (rsync) wrote:
 Hi,
 
 I think the words used are not correct, that's why you (and Kevin)
 are a bit confused. You are in fact asking for incremental
 backup, not a snapshot :
 http://en.wikipedia.org/wiki/Snapshot_(computer_storage). In this
 case, you can use rsync with a few flags to delete (or move in 
 another directory) the modified/deleted files regarding the last
 backup.
 
 This would look something like this ((h)uman, do not (x)cross
 filesystem boundaries, (P)rogress, (H)ard-links, (a)rchive, aka 
 recursive+symlinks+perms+time+owner+group+specials and g(z)ip):
 
 # Cloning your data to a distant folder with timestamp rsync
 -hxPHaz --exclude=exclude_list source target/$NOW/ #
 Incremental backup only rsync -hxPHaz --exclude=exclude_list
 --delete --backup --backup-dir=backup source target/latest/ #
 You can even suffix the modified/deleted files rsync -hxPHaz
 --exclude=exclude_list --delete --backup --backup-dir=backup
 --suffix=~$NOW source target/latest/
 
 Hope that will help, Greg
 
 
 
 
 On Thu, Sep 4, 2014 at 11:21 PM, Kevin Korb k...@sanitarium.net 
 mailto:k...@sanitarium.net wrote:
 
 Yes, you have different scripts now at least.  But I still don't
 see any snapshotting.  The only difference in these is that the
 full one rsyncs to a date+time stamped directory while the
 snapshot one rsyncs to a directory named snapshot.  As far as I
 can tell both will be a complete copy with no relationship to the
 other.
 
 On 09/04/2014 05:17 PM, Chris wrote:
 On Thu, 2014-09-04 at 17:00 -0400, Kevin Korb wrote:
 Do you actually have any snapshots currently?  From the
 scripts you posted it seems to just be rsyncing to the same dir
 every run and only claiming to be making snapshots.  If you do
 have snapshots now then something else is happening in addition
 to this script.
 
 It was pointed out to me in a direct message that I had 
 inadvertently posted the same scripts (full backup) in pastebin. 
 Here are the correct one.
 
 Full http://pastebin.com/dEk7kBip
 
 Snapshot http://pastebin.com/H7SuABN1
 
 
 On 09/04/2014 04:58 PM, Chris wrote:
 On Thu, 2014-09-04 at 12:46 -0400, Kevin Korb wrote:
 The scripts you posted look the same to me.  And I don't
 see any form of snapshotting.  The $NOW variable is set and
 is echoed but it is never actually used.  For an rsync
 snapshot I would expect to see either rsync --link-dest or
 a cp -al depending on the age of the script.
 
 The only reason to need two scripts would be because the 
 snapshot would need 3 dirs to work with (the source, the 
 target, and the previous backup) but that could also be 
 handled with 1 script and a simple existing check.
 
 On 09/04/2014 12:30 PM, Chris wrote:
 I have two scripts that a kind soul on this list wrote
 for me over 4yrs ago. I got to looking at them the other
 day because my old box crashed and had to build a new one
 also got a new backup USB drive since I'm still copying
 over things from the old one. The first one is for a
 full backup:
 
 http://pastebin.com/XF6Zm42A
 
 Works great, does exactly what it's supposed to do. The 
 second is for a 'snapshot' which is where I get a bit 
 confused. I would think that a 'snapshot' would be just
 the changed files either since the last full backup or
 since the last 'snapshot' the night before. It seems
 though that it's actually the same as a full backup. I
 don't profess to be a script person so I have no idea if
 it's doing what it should or something needs to be
 changed. Below is the 2nd script:
 
 http://pastebin.com/MkBzJnux
 
 Any advice would be appreciated.
 
 Chris
 
 
 Thanks Kevin, I guess for now I'll leave them as they are
 until I can get smart on scripting.
 
 Chris
 
 
 
 
 
 -- 

Re: Backup scripts

[Henri Shustak]

Hello,

You may want to check out the LBackup source code as a possible starting point 
if you are looking to create your own customised backup script : 
http://www.lbackup.org/source

Disclaimer : I am involved with the development of LBackup.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org



On 5/09/2014, at 9:17 AM, Chris cpoll...@embarqmail.com wrote:

 On Thu, 2014-09-04 at 17:00 -0400, Kevin Korb wrote:
 Do you actually have any snapshots currently?  From the scripts you
 posted it seems to just be rsyncing to the same dir every run and only
 claiming to be making snapshots.  If you do have snapshots now then
 something else is happening in addition to this script.
 
 It was pointed out to me in a direct message that I had inadvertently
 posted the same scripts (full backup) in pastebin. Here are the correct
 one.
 
 Full
 http://pastebin.com/dEk7kBip
 
 Snapshot
 http://pastebin.com/H7SuABN1
 
 
 On 09/04/2014 04:58 PM, Chris wrote:
 On Thu, 2014-09-04 at 12:46 -0400, Kevin Korb wrote:
 The scripts you posted look the same to me.  And I don't see any
 form of snapshotting.  The $NOW variable is set and is echoed but
 it is never actually used.  For an rsync snapshot I would expect
 to see either rsync --link-dest or a cp -al depending on the age
 of the script.
 
 The only reason to need two scripts would be because the
 snapshot would need 3 dirs to work with (the source, the target,
 and the previous backup) but that could also be handled with 1
 script and a simple existing check.
 
 On 09/04/2014 12:30 PM, Chris wrote:
 I have two scripts that a kind soul on this list wrote for me
 over 4yrs ago. I got to looking at them the other day because
 my old box crashed and had to build a new one also got a new
 backup USB drive since I'm still copying over things from the
 old one. The first one is for a full backup:
 
 http://pastebin.com/XF6Zm42A
 
 Works great, does exactly what it's supposed to do. The second
 is for a 'snapshot' which is where I get a bit confused. I
 would think that a 'snapshot' would be just the changed files
 either since the last full backup or since the last 'snapshot'
 the night before. It seems though that it's actually the same
 as a full backup. I don't profess to be a script person so I
 have no idea if it's doing what it should or something needs to
 be changed. Below is the 2nd script:
 
 http://pastebin.com/MkBzJnux
 
 Any advice would be appreciated.
 
 Chris
 
 
 Thanks Kevin, I guess for now I'll leave them as they are until I
 can get smart on scripting.
 
 Chris
 
 
 
 
 -- 
 Chris
 31.11°N 97.89°W (Elev. 1092 ft)
 16:13:32 up 1 day, 7:44, 1 user, load average: 0.13, 0.27, 0.33
 Ubuntu 14.04 LTS, kernel 3.13.0-35-generic
 
 -- 
 Please use reply-all for most replies to avoid omitting the mailing list.
 To unsubscribe or change options: 
 https://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Beating a dead horse

[Henri Shustak]

 Every year or two I get stuck on this same problem involving
 excluding.

This reply assumes that you would like to backup specific directories via the 
excludes file, which is believe is what you are attempting to accomplish.

Take a look at the information at the following URL : 
http://www.lbackup.org/developer/advanced_excludes_file_examples

Although the URL above is related specifically to the LBackup excludes file, 
the information is relevant to rsync (as LBackup is a wrapper to rsync) and 
outlines one possible solution. If you get stuck or require clarification, then 
please let me know.



This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: silent data corruption with rsync

[Henri Shustak]

 Anyway, if They care about their data , They use checksumming for storing 
 their data on disk, do They ? ;) snip silent bitrot on disks _does_ happen

I totally agree. Storage devices fail and if you need to know if the data is 
the same then a checksum is your best bet. If you want to do your best to 
mitigate this issue to the best of your ability, then there are some good file 
system choices which may be made prior to the spinning platters disks having 
bad blocks or the flash memory failing.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Contents of rsync digest

[Henri Shustak]

 When I try to run a scheduled backup I get the following error:
 
 Error in socket I/O (code 10)
 
 How do I fix this error?


Are you connecting to a remote system to pull or push data? If so are you able 
to ping the remote system or connect to that system on the specified port using 
telnet?

Next, assuming you are running rsync directly to perform the backup, please 
reply to the list with the command you are running and also please provide 
additional context relating to the data which is returned by rsync.

Finally, you could try adding -v to the command line to request more verbose 
information from rsync.



This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Bugg when using Extended Attributes flag -X

[Henri Shustak]

Hello,

One approach is to backup to a disk image on Mac OS X (.sparsebundle) and then 
to push or pull the disk image over to your remote GNU/LINUX system (possibly 
via rsync. LBackup has a scripting sub-system to handle exactly this kind of 
situation. It is not as fancy as the bug fix you proposed. However, it will 
result in your backup having various extended attributes.

This http://www.lbackup.org/initialization_scripts/ URL provides information 
about the initialisation scripts which are designed to make configuring backup 
scenarios like the one listed above (image attaching and detaching) easier. 
Much more work is needed to extend the repository of initialisation scripts 
which are bundled with LBackup. However, the disk image pre and post hook 
initialization script which is listed on that page works a treat and saves 
time. It is just a starting point. Contributions, feedback and alliterative 
approaches to this area of LBackup are certainly welcome. 

The idea behind the initialization scripts is to simplify (via automation) a 
range of steps (often within the LBackup scripting sub-system) so that the 
setup is as painless as possible. However, ideally the initialisation scripts 
could eventually extend well beyond the scripting subsystem and into other 
areas in order to simplify the setup monitoring and verification of backups on 
multiple servers. 

Hope this helps.
Henri



This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

On 5/02/2014, at 12:54 AM, Sun_Blood sbl...@gmail.com wrote:

 Hello,
 
 I found that using rsync on OS X can give some problems when it comes to 
 Extended Attributes (-X flag).
 
 The server I use has Ubuntu with the filesystem XFS and I am trying to backup 
 a OS X system to it. The problem is as far as I understand it that Linux 
 Kernel has a liming on 64k för Extended Attributes and OS X don’t have this 
 limit.
 
 Some error output.
 rsync: rsync_xal_set: 
 lsetxattr(/srv/danne/extern2/1000_EXT/2013/2013-03-05/IMG_6872-Edit.tif,user.com.apple.ResourceFork)
  failed: Argument list too long (7)
 
 Error 2
 rsync: rsync_xal_set: 
 lsetxattr(/srv/nas/home/apple_bak_rsync/x/Pictures/iPhoto 
 Library/Database/BigBlobs.apdb,user.com.apple.FinderInfo) failed: 
 Operation not permitted (1)
 
 Both this errors will go away if removing the -X flag from rsync.
 
 What I would like to see is a feature in rsync that checks the destination 
 operation system environment and if it can’t handle the size of the EA being 
 transferred then stores the EA information in a separate file.
 
 //Sun_Blood
 
 -- 
 Please use reply-all for most replies to avoid omitting the mailing list.
 To unsubscribe or change options: 
 https://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: need help with an rsync patch

[Henri Shustak]

 The solution is not to refuse to backup any file that is a hard link.
 There are legitimate reasons to have hard links and ignoring them
 means you aren't backing up everything.

I agree that preserving hard links may be important in some situation. There 
are certainly legitimate reasons to preserve hard links within a backup.

To more than a couple of years I have been weighing up the advantages and 
disadvantages relating to including a hard link preservation support within 
LBackup. The latest alpha build of LBackup now includes support for hard link 
preservation.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Password from the stdout

[Henri Shustak]

 Use an ssh key instead.

To get started, the following URL offers introductory information : 
http://www.lbackup.org/network_backup#creating_ssh_keys_for_testing

Hope this helps.


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync 3.0.9 partial file left after CTRL-C WITHOUT using --partial

[Henri Shustak]

Hello,

I was unable to reproduce this issue on an OS X system with rsync 3.0.9 
(compiled from source) or 2.6.9 (default on 10.8.2). Output from the testing I 
performed is below.


 snip
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx/src ‹›  13-04-07 - 0:41:08
 ╰─$ dd if=/dev/urandom of=source_file bs=10 count=1
 1+0 records in
 1+0 records out
 10 bytes transferred in 105.312418 secs (9495556 bytes/sec)
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx/src ‹›  13-04-07 - 0:43:17
 ╰─$ du -hs ./source_file 
 954M  ./source_file
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx/src ‹›  13-04-07 - 0:43:25
 ╰─$ cd ..
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:43:31
 ╰─$ mkdir dst
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:43:36
 ╰─$ rsync -a ./src/ ./dst 
 ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at 
 /SourceCache/rsync/rsync-42/rsync/rsync.c(244) [sender=2.6.9]
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:44:26
 ╰─$ ps -a | grep rsync
   20 ↵
  2181 ttys0000:00.00 grep rsync
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:44:37
 ╰─$ ls -la ./dst 
 total 0
 drwxr-xr-x  2 henri  wheel   68 Apr  7 00:44 .
 drwxr-xr-x  4 henri  wheel  136 Apr  7 00:43 ..
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:45:03
 ╰─$ rsync -a ./src/ ./dst
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:45:59
 ╰─$ ls -la ./dst 
 total 1953128
 drwxr-xr-x  3 henri  wheel 102 Apr  7 00:41 .
 drwxr-xr-x  4 henri  wheel 136 Apr  7 00:43 ..
 -rw-r--r--  1 henri  wheel  10 Apr  7 00:43 source_file
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:46:03
 ╰─$ rm ./dst/source_file 
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:46:15
 ╰─$ rsync -a ./src/ ./dst
 ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at 
 /SourceCache/rsync/rsync-42/rsync/rsync.c(244) [sender=2.6.9]
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:46:39
 ╰─$ rsync -a ./src/ ./dst 
   20 ↵
 ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at 
 /SourceCache/rsync/rsync-42/rsync/rsync.c(244) [sender=2.6.9]
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:46:54
 ╰─$ ps -a | grep rsync
   20 ↵
  2280 ttys0000:00.00 grep rsync
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:46:59
 ╰─$ ls -la ./dst 
 total 0
 drwxr-xr-x  2 henri  wheel   68 Apr  7 00:46 .
 drwxr-xr-x  4 henri  wheel  136 Apr  7 00:43 ..
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:47:03
 ╰─$ 
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:47:39
 ╰─$ rsync -a --partial ./src/ ./dst
 ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at 
 /SourceCache/rsync/rsync-42/rsync/rsync.c(244) [sender=2.6.9]
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:48:00
 ╰─$ ls -la ./dst  
   20 ↵
 total 402432
 drwxr-xr-x  3 henri  wheel102 Apr  7 00:48 .
 drwxr-xr-x  4 henri  wheel136 Apr  7 00:43 ..
 -rw-r--r--  1 henri  wheel  206045184 Apr  7 00:48 source_file
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:48:05
 ╰─$ rm ./dst/source_file 
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:50:02
 ╰─$ rsync -a --partial --inplace ./src/ ./dst
 ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at 
 /SourceCache/rsync/rsync-42/rsync/rsync.c(244) [sender=2.6.9]
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:50:48
 ╰─$ ls -la ./dst  
   20 ↵
 total 1065472
 drwxr-xr-x  3 henri  wheel102 Apr  7 00:50 .
 drwxr-xr-x  4 henri  wheel136 Apr  7 00:43 ..
 -rw---  1 henri  wheel  545521664 Apr  7 00:50 source_file
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:50:54
 ╰─$ 
 
 snip
 
 ╭─henri@mac  /tmp/test_rsync_3.0.9_osx ‹›  13-04-07 - 0:55:56
 ╰─$ /usr/local/bin/rsync_v3.0.9 -a ./src/ ./dst 
 ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(657) 
 [sender=3.0.9]
 ╭─henri@mac  /tmp/test_rsync_3.0.9_osx ‹›  13-04-07 - 0:56:46
 ╰─$ ls -la ./dst  
   20 ↵
 total 0
 drwxr-xr-x  2 henri  wheel   68 Apr  7 00:56 .
 drwxr-xr-x  4 henri  wheel  136 Apr  7 00:43 ..
 ╭─henri@mac  /tmp/test_rsync_3.0.9_osx ‹›  13-04-07 - 0:56:53
 ╰─$ /usr/local/bin

Re: rsync 3.0.9 partial file left after CTRL-C WITHOUT using --partial

[Henri Shustak]

Hello,

Just to clarify is the file(s) only left when using rsync to transfer to a 
non-root partition? The reason I ask is that the tests performed (as quoted in 
the previous email where carried out on the root '/' partition). I can easily 
retry on an another volume (network mount or locally attached FW HDD / USB 
memory stick).

Thanks.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org


 Hello,
 
 I was unable to reproduce this issue on an OS X system with rsync 3.0.9 
 (compiled from source) or 2.6.9 (default on 10.8.2). Output from the testing 
 I performed is below.
 
 
 snip
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx/src ‹›  13-04-07 - 0:41:08
 ╰─$ dd if=/dev/urandom of=source_file bs=10 count=1
 1+0 records in
 1+0 records out
 10 bytes transferred in 105.312418 secs (9495556 bytes/sec)
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx/src ‹›  13-04-07 - 0:43:17
 ╰─$ du -hs ./source_file 
 954M ./source_file
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx/src ‹›  13-04-07 - 0:43:25
 ╰─$ cd ..
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:43:31
 ╰─$ mkdir dst
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:43:36
 ╰─$ rsync -a ./src/ ./dst 
 ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at 
 /SourceCache/rsync/rsync-42/rsync/rsync.c(244) [sender=2.6.9]
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:44:26
 ╰─$ ps -a | grep rsync   

 20 ↵
 2181 ttys0000:00.00 grep rsync
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:44:37
 ╰─$ ls -la ./dst 
 total 0
 drwxr-xr-x  2 henri  wheel   68 Apr  7 00:44 .
 drwxr-xr-x  4 henri  wheel  136 Apr  7 00:43 ..
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:45:03
 ╰─$ rsync -a ./src/ ./dst
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:45:59
 ╰─$ ls -la ./dst 
 total 1953128
 drwxr-xr-x  3 henri  wheel 102 Apr  7 00:41 .
 drwxr-xr-x  4 henri  wheel 136 Apr  7 00:43 ..
 -rw-r--r--  1 henri  wheel  10 Apr  7 00:43 source_file
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:46:03
 ╰─$ rm ./dst/source_file 
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:46:15
 ╰─$ rsync -a ./src/ ./dst
 ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at 
 /SourceCache/rsync/rsync-42/rsync/rsync.c(244) [sender=2.6.9]
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:46:39
 ╰─$ rsync -a ./src/ ./dst

 20 ↵
 ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at 
 /SourceCache/rsync/rsync-42/rsync/rsync.c(244) [sender=2.6.9]
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:46:54
 ╰─$ ps -a | grep rsync   

 20 ↵
 2280 ttys0000:00.00 grep rsync
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:46:59
 ╰─$ ls -la ./dst 
 total 0
 drwxr-xr-x  2 henri  wheel   68 Apr  7 00:46 .
 drwxr-xr-x  4 henri  wheel  136 Apr  7 00:43 ..
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:47:03
 ╰─$ 
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:47:39
 ╰─$ rsync -a --partial ./src/ ./dst
 ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at 
 /SourceCache/rsync/rsync-42/rsync/rsync.c(244) [sender=2.6.9]
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:48:00
 ╰─$ ls -la ./dst 

 20 ↵
 total 402432
 drwxr-xr-x  3 henri  wheel102 Apr  7 00:48 .
 drwxr-xr-x  4 henri  wheel136 Apr  7 00:43 ..
 -rw-r--r--  1 henri  wheel  206045184 Apr  7 00:48 source_file
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:48:05
 ╰─$ rm ./dst/source_file 
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:50:02
 ╰─$ rsync -a --partial --inplace ./src/ ./dst
 ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at 
 /SourceCache/rsync/rsync-42/rsync/rsync.c(244) [sender=2.6.9]
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:50:48
 ╰─$ ls -la ./dst 

 20 ↵
 total 1065472
 drwxr-xr-x  3 henri  wheel102 Apr  7 00:50 .
 drwxr-xr-x  4 henri  wheel136 Apr  7 00:43 ..
 -rw---  1 henri  wheel  545521664 Apr  7 00:50 source_file
 ╭─henri@mac  /tmp/test_rsync_2.6.9_osx ‹›  13-04-07 - 0:50:54
 ╰─$ 
 
 snip
 
 ╭─henri@mac  /tmp/test_rsync_3.0.9_osx ‹›  13-04-07 - 0:55:56
 ╰─$ /usr/local/bin/rsync_v3.0.9 -a ./src/ ./dst

Re: Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

[Henri Shustak]

You may be interested in having a look at LBackup http://www.lbackup.org, an 
open source (released under the GNU GPL) backup system.

Essentially, LBackup is a wrapper for rsync. If you are working on your own 
script. Feel free to look at how LBackup works (primely written in bash at 
present) and use the features, ideas or even code snippets within your project 
in order to make your project even better than LBackup.

All the best with your project. Looking forward to seeing what you come up with.



This email is protected by LBackup, an open source backup solution
http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Time rsYnc Machine (tym)

[Henri Shustak]

 Push is when you run your backup program (rsync and whatever script)
 on the machine being backed up and you push/upload your data to the
 backup system.
 
 Pull is when you run your backup program on the backup system and
 pull/download the data from the machine being backed up.

You may also find the following link of interest : 
http://www.lbackup.org/network_backup_strategies


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync based on checksum only

[Henri Shustak]

 Wow!  Thanks for making it so easy.  I will try that asap.

If you do not have any luck with the patched version of rsync there are various 
projects which spring to mind which offer this kind of functionality. 

However, I would suggest that rsync is the most stable project I have ever seen 
for this kind of coping. 

As such, if the patch approach works I would stick with rsync. If you have 
problems with the patch then report back to the list as rsync is an very 
reliable and mature tool with some great developers and people on this list to 
assist you.

Hope this helps.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 8712] --link-dest doesn't work if target file exists (but needs updating)

[Henri Shustak]

 I'm also hit by this issue.  Is there any prediction as to whether and, if so,
 when something is going to happen? :)

Until such time as the situation with --link-dest changes consider syncing into 
an empty directory (if at all possible). 

If you are using rsync for backup then you may wish to take a look at LBackup 
http://www.lbackup.org or the source code http://www.lbackup.org/source/.

At present LBackup will only keeps a single copy of the previous backup attempt 
and this attempt is not referenced or used in any way to reduce bandwidth for 
subsequent backup attempts. This approach may change in a future release of 
LBackup (eg. keeping a history of previous failed backup attempts and adding 
previous failed backup attempts as link-dest paths). However, at present this 
is simple approach is employed to ensure that LBackup is generating a reliable 
backup snapshot(s).

In the mean time, hopefully, either LBackup or the source code for the project 
will be of assistance. 

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --link-dest isn't linking

[Henri Shustak]

 rsync -Ha --link-dest=/media/4tb/bak/panic-2012-01-01 
 /media/2tb/bak/panic-2012-02-01 /media/4tb/bak/

It seems that you may be attempting to hard-link between two file systems. One 
is mounted as /media/4tb the other is mounted as media/2tb. 

My understanding is that it is not normally possible to hard-link between file 
systems. 

The following is quoted from the man page for the 'ln' command :
 Hard links may not normally refer to directories and may not span file 
 systems.


Hopefully this information is helpful.


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --link-dest doesn't work if target file exists (but needs updating)

[Henri Shustak]

 On 12-01-20 06:01 PM, Kevin Korb wrote:
 Someone has requested it:
 https://bugzilla.samba.org/show_bug.cgi?id=7870
 
 I'm not really sure that is the same bug.  Maybe it is.  Not convinced
 though.  I guess I can file my own bug and ask 7870's OP to see if it's
 the same issue.
 
 But is seems like an rm -rf and a cp -al would do the same thing.
 
 But as I said before, it won't.  Not all files in the destination are
 the same as the source and files may exist in the destination that don't
 exist in the source.


Is what you are trying to achieve part of a backup system / script?


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: securing rsync backup solutions?

[Henri Shustak]

 - - laptop-to-server: the laptop tends to roam a lot, network conditions
 vary from place to place, and the rsync will be initiated manually, so
 it is likely to be a `push' operation, the ssh keys will definitely need
 to be encrypted and unlocked manually, but even with that security, I
 want to ensure the keys have the minimum level of access possible on the
 server - the laptop definitely shouldn't have a key for the root account

You may be interested in the approach listed at the following URL : 
http://www.lbackup.org/synchronizing_disk_images_between_machines

Unfortunately, the URL above is aimed specifically towards Mac OS X systems. 
However, I am hopeful that there is a similar (band based) disk image system 
available for GNU/LINUX. If you find something like that then please let me 
know.

Using a virtual file system allows you to save permissions within the disk 
image and then push the changes within the disk image to a remote server 
without requiring root access to the remote server.

Finally, Kevin's comments regarding push and pull backups are worth keeping in 
mind.


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org





-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Raise an error if file exists on destination

[Henri Shustak]

 I am trying to use rsync to copy files from several origins to a single 
 destination. I would like to detect when a file gets overwritten because it 
 is present with the same relative path/name in several origins:
 
 
 Origin1:
 foo/bar.txt
 
 Origin2:
 foo/bar.txt
 
 Destination after copy:
 foo/bar.txt - is that from Origin1 or origin2?
 
 
 I know I can use the `--ignore-existing` option to make sure only the first 
 version is kept, or I could remove this option to keep the last one. But I 
 would like to be notified when this happens, either with a warning or an 
 error.
 
 Is that possible with rsync?


The -i --itemize-changes, output a change-summary for all updates may do what 
you want.

However, my understanding for files which are detected to be the same in the 
source(s) and the destination will not be updated / overwritten. Perhaps 
someone else on the rsync mailing list may be able to clarify this point or 
answer your question more appropriately.



 This email is protected by LBackup, 
 an rsync based, open source backup system
 designed for use with Mac OS X and GNU/Linux
 http://www.lbackup.org





-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Mac OS X : get_xattr_names: llistxattr(some/path/here, 1024) failed error

[Henri Shustak]

 Hi,
 
 I'm trying to make a small script to get rid of Apple's TimeMachine. The aim 
 is to backup the files of my company.
 I setup a MacMini with a lot of storage attached to it. The MacMini connects 
 every once in a while to our data server (XServe) through SSH and pulls the 
 files that need to be saved (I'm using the --link-dest option to limit the 
 amount of files transferred, and to keep several backups).
 
 I've followed Mike Bombich's instructions to compile rsync 3.0.7 with patches 
 (fileflags, crtimes, crtimes-64bit, crtimes-hfs+ and hfs_compression) on both 
 machines. I'm running the script as root with the following options (which 
 are supposed to do the work) :
 
-aNHAXx --fileflags --force-change --protect-decmpfs --numeric-ids
 
 The script works pretty fine except for one thing. When rsync builds the file 
 list, I get hundred of errors such as :
 
rsync: get_xattr_names: llistxattr(some/path/here,1024) failed: No such 
 file or directory (2)
 
 Please notice that these errors don't appear if I'm not using the --link-dest 
 option or if the linked-dest-directory doesn't exist (first backup).
 Also, some/path/here does exist on both side.
 
 I've read some doc about this error but it doesn't seem to fit my case and 
 I'm kinda stuck with it. It doesn't seem to be a serious issue but I'd like 
 to be sure about it and know if there is a way to get rid of them.
 
 Do you guys have any clue of what's going wrong or what I may doing wrong ?

As a first step you could try removing the -A --acls, preserve ACLs (implies 
--perms) option. It could be that the source system and your system do not 
have the same users.

Also, based upon the error you are receiving, one other option to consider 
removing would be the -X --xattrs preserve extended attributes. 

It is possible, that something else is causing the error which you are seeing. 
Hopefully, at least one other person on this list will have some ideas which 
you may find helpful.

As a starting point try removing the arguments listed above and report back.



 This email is protected by LBackup, 
 an rsync based, open source backup system
 designed for use with Mac OS X and GNU/Linux
 http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 8566] Spotlight comments (extended attributes) are not synced

[Henri Shustak]

 I downloaded 3.0.9 source + patches, patched in the 2 patches:
 patch -p1 patches/fileflags.diff
 patch -p1 patches/crtimes.diff
 And although running with the appropriate arguments (as in the original
 report), the comments still do not get synced!
 
 Help please!


I believe this issue has been previously mentioned. I do not remember the 
outcome or if it has been resolved. There may be a duplicate bug in the 
tracking system or something mentioned on the mailing list relating to this 
bug. I would suggest having a hunt in the mailing list archives and the bug 
tracking system. 

A quick search brings up the following (unfortunately no answers).
http://www.mail-archive.com/rsync@lists.samba.org/msg21846.html

If you use the latest development branch is the problem still present?

Hope this helps.


-
This message is protected by LBackup, an open source backup solution.
http://www.lbackup.org

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Replacing rsync on OSX servers.

[Henri Shustak]

 Having built and tested 3.0.7 and ready to send it out into production, can 
 anybody point me to 'best practices' for updating the binary and man pages 
 and other issues around upgrading from the dodgy v2.6.9 that ships with 
 late-10.4-thru-10.7?


You could use /usr/local/bin as the install location for the rsync binary and 
then update your environment variable accordingly. This is just one possible 
approach.

 Not having replaced an OSX tool in the past, I've got no idea if Apple is 
 prone to update these utils, or just 'fix' what I've done automatically 
 during routine checks and system upgrades. Any experiences would be 
 appreciated.

If you replace the binary then it is possible that an Apple would overwrite 
your files. 

Despite this message not directly answering your questions, hopefully you find 
some of the information I have provided useful / informative.


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Strange bug (buffer overflow) happening only under cron

[Henri Shustak]

Hi Pedro,


 Mmh, I put uname -a into the cron job and it says:
 
 core file size  (blocks, -c) 0
 data seg size   (kbytes, -d) unlimited
 file size   (blocks, -f) unlimited
 max locked memory   (kbytes, -l) unlimited
 max memory size (kbytes, -m) unlimited
 open files  (-n) 2560
 pipe size(512 bytes, -p) 1
 stack size  (kbytes, -s) 8192
 cpu time   (seconds, -t) unlimited
 max user processes  (-u) 266
 virtual memory  (kbytes, -v) unlimited
 
 which is the same (exactly) as the output when run from
 the shell (bash, by the way, in both environments).
 
 Thanks, anyway.


Have you tried using LaunchD as you are scheduling from Mac OS X?



---
This email is protected by LBackup
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync connection/transfer error

[Henri Shustak]

Hello,

Is your output on this problematic network similar to the following : 

$ telnet mirrors.usc.edu 873
Trying 68.181.195.4...
Connected to hpc-mirror.usc.edu.
Escape character is '^]'.
@RSYNCD: 30.0

Also, what happens if you add -n (--dry-run) to the following command :
 rsync -artv rsync://mirrors.usc.edu/centos/5.6/updates/i386/RPMS/ 
 --exclude=debug/ /var/www/html/centos/5/updates/i386

If it is working on from networks (which it seems to be) then I suspect there 
is a problem with the connection.

Is the firewall opened for connections on port 873 to all systems on the 
internet? The reason I ask is because you could start an iperf 
http://iperf.sourceforge.net/ server (operating on port 863) on a remote 
system and see if connections works from the problematic network/host and what 
kind of throughput you are getting. Perhaps the connection opens but once data 
starts being passed something is going wrong? 

The other thing is that you could setup an rsync server on a different system 
and see if it has the same problem. With such a test you should see the same 
issue if it is a network issue.

As far as I am able to see the mirror you are connecting to for the source 
seems to be working fine mirrors.usc.edu. I suspect there is an issue with 
the network connection or host. 

Check the environment variables on the system for anything which may be causing 
issues. Also, you if possible you could try building a copy of rsync from 
source and see if that helps?

Finally, have you attempted the same rsync command from another system within 
the problematic network? If so what was the result did it manage to pull the 
data down from the server?

Hopefully, at least some of the information above is helpful =:^)


--
This email is protected by LBackup
http://www.lbackup.org






-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync connection/transfer error

[Henri Shustak]

Hello,

It seems that the issue is todo with the network. However, I would suggest you 
try with a different computer system to be sure. Additional notes follow :

--

 server: iperf -s -D -p 873
 (problematic network) client: iperf -c iperf.server.com -L 873
  
 The client times out.  I know that iperf is running on the server because I 
 can
  
 telnet iperf.server.com 873
  
 and get connection.  So, I'm not sure if this is the correct process? 

If you remove the -D option then you should see output logged directly to the 
console on the iperf server. Adding the --format M may make the output easier 
to read. However neither of these are issues which would stop this from working 
IMHO.

 I'm told that the firewall is open for bi-directional connections on port 873 
 to all systems on the Internet. 

Rather than trying a bi-directional connection you could try the following on 
the client :
iperf -c iperf.server.com -p 873

Assuming you are behind NAT, this will confirm that a connection may be 
established from your client to the server and that data transfer is possible. 

 The team the runs the network is notorious for needed very concrete evidence 
 for making firewall changes so I need to build my case ;-)..

I would suggest the iperf command I have listed above and seeing if that works. 
If you have a system (eg. Laptop/Netbook) which you could easily move between 
networks. Then you could try from outside of the problematic network (eg. a 
network which outbound TCP traffic on port 873 will be allowed) and then from 
within within the problematic network.

If the problematic network is not working and the iperf / rsync is working from 
a known good network. Then consider lending the laptop to the network team and 
ask them to make changes and provide them a system which they may use to test 
any changes with?

Hope this helps.


--
This email is protected by LBackup
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: With rsync --link-dest, is it possible to determine which files are new?

[Henri Shustak]

 SHA1

You may also find fingerprint to be a useful tool. However, it will require 
ruby.

---
This email is protected by LBackup
http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Estimating backup usage with dir-merge filter

[Henri Shustak]

 It sounds like you missed the point of Kevin's message (in the other fork of 
 this thread).  The point wasn't to use
 `du`, it was that you can run your stats against the backed-up files, not 
 the source.  Then you're only running stats
 against the results of running the backup using the filters, so you don't 
 need to filter them again.
 
 I got that but neglected to respond to the whole group.  My mistake.
 The backups are being performed using BackupPC to a central server
 where compression and de-duplication is done.  While it's true that
 the actual storage on the backup server being consumed by each user is
 less because of these, I don't have any problem hiding this from them
 and instead telling them what their uncompressed and duplicated usage
 is instead.  It has more of an effect that way if you know what I
 mean.
 
 If that doesn't make sense or isn't possible (backups are on some remote 
 server), then just use your rsync command
 with '--list-only', and post-process that list.
 
 I've been tinkering with using --verbose and --dry-run then parsing
 the total size our of the last line of the output and I think I'm
 close.  Curiously, when I don't include the --filter option as a
 baseline, I'm not getting the same results as du.
 
 $ du -sb . | awk '{print $1}'
 508625653
 
 $ rsync --dry-run --verbose -a . /tmp/does_not_exist | tail -1 | awk
 '{print $4}'
 506037893
 
 The difference is minimal and probably negligible for this purpose but
 I'm still curious where it's coming from.  Maybe there are some sparse
 files in there somewhere.

Do you have the same discrepancy if you use the --stats option?



 This email is protected by LBackup
 http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 8456] improve --link-dest bahaviour

[Henri Shustak]

 In general - --link-dest works as expected *only*, if the destination 
 directory
 is empty.

Going along with this idea of ensuring the destination directory is empty. 
LBackup is a rsync  backup wrapper system which may be of assistance in this 
regard.


---
This email is protected by LBackup
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 8450] New: --link-dest seems not to work mounted NTFS file systems

[Henri Shustak]

Just to clarify, Is the destination file system NTFS?

Also, is this storage directly attached or is it mounted via the network?

Finally, you mentioned it is mounted read-only would you please clarify why the 
drive you are writing to is mounted in this way? 

Thanks.
Henri 


This email is protected by LBackup
http://www.lbackup.org


 https://bugzilla.samba.org/show_bug.cgi?id=8450
 
   Summary: --link-dest seems not to work mounted NTFS file
systems
   Product: rsync
   Version: 3.0.8
  Platform: x86
OS/Version: Linux
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
AssignedTo: way...@samba.org
ReportedBy: toralf.foers...@gmx.de
 QAContact: rsync...@samba.org
 
 
 I use rsync to create daily/weekly/monthly backups to an external USB drive of
 my Gentoo Linux using this command line options :
 
 --archive --delete --delete-excluded --exclude-from=/exclude.list
 --link-dest=old dest --stats --verbose src dest
 
 This works like a charm for all files located on my ext3 Linux partition.
 However for the 2 files of a NTFS drive not.
 
 Here rsync always transfers an unchanged file too, even if old dest contains
 that file (from a backup made yesterday) and therefore it should better be
 hard-linked only instead of being transferred twice.
 
 What is really curious - at the beginning of using this backup solution (some
 weeks ago) the dest doesn't contain an older version of that file, and there
 hard-linking worked IIRC.
 
 The NTFS drive is mounted read-only:
 
 tfoerste@n22 ~ $ mount | grep C
 /dev/sda1 on /mnt/C type ntfs
 (ro,noexec,nosuid,nodev,noatime,uid=1000,gid=100,umask=0027)
 
 src  is /mnt/C/notes/data/archive/
 dest is /mnt/media/daily/f/
 old dest is /mnt/media/weekly/b (given as ../../weekly/b at the command 
 line)
 
 Here you can see that different hard links are used :
 
 tfoerste@n22 ~ $ ls -il /mnt/C/notes/data/archive/l_DE1448.nsf
 /mnt/media/daily/f/archive/l_DE1448.nsf 
 /mnt/media/weekly/b/archive/l_DE1448.nsf 
  80719 -r-xr-x--- 1 tfoerste users 8126464 Sep  2 13:40
 /mnt/C/notes/data/archive/l_DE1448.nsf
 2293796 -r-xr-x--- 1 tfoerste users 8126464 Sep  2 13:40
 /mnt/media/daily/f/archive/l_DE1448.nsf
 2310874 -r-xr-x--- 1 tfoerste users 8126464 Sep  2 13:40
 /mnt/media/weekly/b/archive/l_DE1448.nsf
 
 md5sum proves identical content, the file attributes shouldn't changed too
 (NTFS is always mounted read-only, never read-write) :
 
 tfoerste@n22 ~ $ md5sum /mnt/C/notes/data/archive/l_DE1448.nsf
 /mnt/media/daily/f/archive/l_DE1448.nsf 
 /mnt/media/weekly/b/archive/l_DE1448.nsf 
 88ef0d516c5696bf120d749f4b1fc9ba  /mnt/C/notes/data/archive/l_DE1448.nsf
 88ef0d516c5696bf120d749f4b1fc9ba  /mnt/media/daily/f/archive/l_DE1448.nsf
 88ef0d516c5696bf120d749f4b1fc9ba  /mnt/media/weekly/b/archive/l_DE1448.nsf
 
 OTOH does it make any difference whether the dest drive at the external USB
 drive is mounted as ext2 instead of ext3 ?
 
 -- 
 Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
 --- You are receiving this mail because: ---
 You are the QA contact for the bug.
 -- 
 Please use reply-all for most replies to avoid omitting the mailing list.
 To unsubscribe or change options: 
 https://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 8456] improve --link-dest bahaviour

[Henri Shustak]

 I'd like to use rsync as an efficient (== do not store the same file twice at
 the backup media) backup solution. The backup should be made into N remote
 directories (rotating each day) _without_ the need to delete the remote
 directory before.

This is essentially, what LBackup is doing. However, keep in mind that if the 
transfer is interrupted. The incomplete backup will be deleted and then it will 
be restarted only using the latest successful backup (with regards link-dest). 
LBackup will always backup to a clean (empty) backup destination directory 
called Section.inprogress.

The LBackup project is open source. As such, you are free to have a look at the 
code to get an idea of what it is doing or even fork the project (github - 
makes this easy) for your specific requirements.

Also, keep in mind that if a file is not hard linked to save space and it has 
been updated then it will end up taking up additional space on the destination 
backup system. 

I have not experimented with de-duplicating file systems. However, if you want 
to save on space then this may be an avenue to investigate as well with regards 
backups, regardless of wheather you use LBackup, a different wrapper, rsync 
directly or some other system.

I hope this helps. 


---
This email is protected by LBackup
http://www.lbackup.org

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync and encryption

[Henri Shustak]

  Which operating system are you running on the system which currently has 
 your personal documents?
 
 
 Ubuntu/Debian

Do you know if there is a virtual file system encryption system which breaks 
the data into bands? I know that Mac OS X has support for breaking an encrypted 
disk image into bands (eg 8MB each). However, I am not sure about for LINUX. I 
have looked at TrueCrypt but was unable to see such an option when I last 
checked.

If you are able to break the virtual file system into bands then keeping this 
in sync on a remote server is far easier. 

The link below has details regarding this and with a few changes it would work 
on a LINUX system.

http://www.lbackup.org/synchronizing_disk_images_between_machines


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Brandysnap -- a new rsync-based snapshot management script

[Henri Shustak]

 Hi Chris,
 
 https://github.com/StarsoftAnalysis/brandysnap
 
 I am involved with the LBackup project. Would you be okay with a link being 
 generated to the Brandy on github page from the LBackup 
 alternativeshttp://www.lbackup.org/alternatives  page?
 
 Yes, that would be great.
 
Done.

 Depending upon the license you release Brandysnap under, perhaps in the 
 future Brandysnap functionality could be incorporated into the LBackup 
 project?
 
 It's GPL'd -- share and enjoy!

Okay. GPL, same as LBackup.

Also, if this is helpful for anyone else on the list. My initial testing of 
BrandySnap shows that it works on Mac OS X. All that was required was an update 
to perl and the installation of some modules via CPAN.

Seems like a great project!

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org (back up and running shortly =:^)


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Brandysnap -- a new rsync-based snapshot management script

[Henri Shustak]

Hi Chris,

 https://github.com/StarsoftAnalysis/brandysnap

I am involved with the LBackup project. Would you be okay with a link being 
generated to the Brandyon github page from the LBackup alternatives 
http://www.lbackup.org/alternatives page?

Depending upon the license you release Brandysnap under, perhaps in the future 
Brandysnap functionality could be incorporated into the LBackup project?

Sounds like a good project. Thank you for posting the link and explanation.

Keep in touch.
Henri

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Using rsync as an incremental backup

[Henri Shustak]

 I'm using rsync to do an incremental backup of my desktop here, to a
 remote server as follows:
 
 #/usr/bin/bash
 
 old=$(date -d 'now - 1 week' +%Y-%m-%d)
 new=$(date +%Y-%m-%d)
 
 rsync -avP --delete --link-dest=../$dir /home/bakers
 bak...@perturb.org:/home/bakers/backup/$new/
 
 This is actually working GREAT! The only problem is that sometimes the
 cronjob won't complete (internet is down, something like that). When it
 tries to run the next week it does --link-dest against a dir that
 doesn't exist. It happily complies and transfers EVERY file because
 there is no source to hardlink from.
 
 I'd really like rsync to exit and throw an error if the --link-dest
 isn't present. I can't find anything in the man page about any fancy
 --list-dest options. Am I missing it?

Hopefully, Scott's suggestion will solve this issue for you.

You may also be interested in the way LBackup works for hard-linked incremental 
backups. It uses a numbering system rather than a date system. 

If you still want to have the date named entries within the file system, it is 
just a matter of enabling a post-action which keeps a date-named symbolic links 
directory to these backup snapshots up to date.

Details on the LBackup post and pre actions are available from the following 
URLs :

  - http://www.lbackup.org/pre_and_post_actions (basic intro on LBackup 
pre/post actions)
  - http://bit.ly/lIMfAo (source code for the example post action script 
mentioned) 

This is all just another way of approaching this issue which you may find of 
interest. The one issue is that the script (source code link above) will need 
to be run on a system which has access to the file system which stores the 
snapshots. It would require modification if you wanted to run it on a remote 
system.

Hope this helps :^)


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Being VERY careful while using the --delete option

[Henri Shustak]

 On UNIX, I am  executing an rsync command, from within a script. The command 
 goes something like this:
 
 /usr/bin/rsync --verbose  --progress --stats --compress --recursive --times 
 --perms --links  --safe-links source_dir/ 
 user@target_machine:/parent_path/source_dir  
 
 In other words, I am replicating source_dir on a remote machine. It ends up 
 next to a lot of sibling, directories, like this:
 
 On target_machine
 
 /parent_path/source_dir
 /parent_path/source_dir2
 
 /parrent_path/source_dirn
 
 Here's my issue. I'd like to use the --delete option in the command above. 
 However, I wake up in a cold sweat worrying about supplying the empty 
 string for source_dir, and thus removing souce_dir, and all it's siblings on 
 target_machine.
 
 Now I realize, I can just be extra careful that the variable pointing to 
 source_string is never empty, but I would like to have other fail safe 
 mechanisms in place.
 
 I hope I have been sufficiently specific to elicit some suggestions. I guess 
 I am looking for the software equivalent of lots of padlocks on the apartment 
 door.

You may wish to consider the use of the -n option. The --dry-run option 
performs a trial run with no changes made. Using this option will allow you to 
check what will happen prior to actually moving ahead.

Just a possibility?


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org





-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: thank you wayne, andrew, paul and everyone else

[Henri Shustak]

+2! :)

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: files not moved immediately to final destination from temp location after rsync returns with success

[Henri Shustak]

 I am using rsync version 3.0.7 on an arm linux based embedded device. The 
 device pulls data periodically from a rsync server and stores the files on an 
 SD card. The partial, temp and final rsync destinations all reside on the SD 
 card.
 
 I came across an issue where it seems that the rsync call returned with a 
 success but the files that it pulled are not moved immediately to its final 
 destination. 


You could try issuing the 'sync' command? However, I do not think believe that 
this should be required. 

Perhaps someone else on this list will have some other ideas?



 This email is protected by LBackup
 http://www.lbackup.org





-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync not reporting diskfull error

[Henri Shustak]

 If pushing data (e.g. a local copy or copy from local to remote), there is a 
 failure where the receiver can try to report an error, die, and the sender 
 gets the error trying to write to the receiver before it gets the error 
 message from around the horn (it would have gone to the generator, and then 
 to the sender if given a chance).  This is something that I've fixed in rsync 
 3.1.0dev (the latest git on the master branch), which now has a mode where 
 the receiver just chews through data waiting for a die signal instead of 
 dying before the error messages can be finished up.  It also supports the 
 --msgs2stderr option, which works for most local and remote-shell situations 
 (but not daemon connections) where any of the 3 processes can output its 
 message to stderr and bypass all the data being sent on the protocol socket.
 
 The upcoming 3.0.8 release won't have either of these items, but 3.1.0 should 
 be released not too long after 3.0.8.

You are a star!

Thank you!




This email is protected by LBackup
http://www.lbackup.org




-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync not reporting diskfull error

[Henri Shustak]

I am involved with the development of lbackup. This message to the rsync 
mailing list is related to the following thread on the lbackup-disccussion 
mailing list : http://tinyurl.com/lbackup-discussion-diskfull

Essentially, I am curious to if any one using rsync 3.0.7 on Mac OS (10.6) 
Server has experienced an out of disk space error and not had a message similar 
to the following reported :

 rsync: mkstemp 
 /Volumes/backup_volume/backups/Section.inprogress/to_backup/file_for_which_there_is_not_enough_space.n3pq0e
  failed: No space left on device (28)
 rsync_v3.0.7(33235) malloc: *** error for object 0xa: pointer being freed was 
 not allocated
 *** set a breakpoint in malloc_error_break to debug
 rsync: connection unexpectedly closed (22413 bytes received so far) [sender]
 rsync error: error in rsync protocol data stream (code 12) at io.c(601) 
 [sender=3.0.7]


Instead have you only seen something similar to the following : 

 rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken 
 pipe (32)
 rsync: connection unexpectedly closed (3225054 bytes received so far) [sender]
 rsync error: error in rsync protocol data stream (code 12) at io.c(601) 
 [sender=3.0.7]


I am attempting to reproduce the message above with no luck and was wondering 
if anyone on this list has seen this kind of message without the No space left 
on device (28) message.


--
This email is protected by LBackup
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Protocol stream error on extended attribute, silent failure to copy all attributes

[Henri Shustak]

 I'm using rsync 3.0.7 on Mac OS X 10.6, compiled according to Mike
 Bombich's instructions at http://www.bombich.com/rsync.html. Rsync
 repeatedly exits with a protocol data stream error when trying to copy
 some com.apple.FinderInfo extended attributes. While testing this issue,
 I found that rsync is not actually copying all extended attributes even
 when there is no error message. I'm using a folder of fonts as an
 example, but I have experienced the protocol error when copying other
 data. This seems like a huge bug, and in my experience those often turn
 out to be operator error. Apologies if I waste anyone's time.
 
 In this example, I'm using rsync to make a copy from scratch of the
 source data: rsync307 -aX --delete Licensed\ Fonts /Volumes/Storage/
 
 I repeatedly get this error:
 
 [sender] internal abbrev error on Licensed Fonts/Postscript/bradley
 (com.apple.FinderInfo, len=32)!
 rsync error: error in rsync protocol data stream (code 12) at
 xattrs.c(636) [sender=3.0.7]
 
 If you repeat the command a few times, sometimes the error is:
 
 rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]:
 Broken pipe (32)
 rsync: connection unexpectedly closed (16827 bytes received so far)
 [sender]
 rsync error: error in rsync protocol data stream (code 12) at io.c(601)
 [sender=3.0.7]
 
 In all my testing, I have only seen the error occur with a
 com.apple.FinderInfo attribute, and only for a directory.
 
 Running rsync with sudo makes no difference.
 
 The system has no problem reading the extended attribute from the source
 folder, or using the xattr command to write it to the target folder:
 
 matt$ xattr -l Licensed\ Fonts/Postscript/bradley/
 com.apple.FinderInfo:
   03 99 01 5D 04 7C 02 2F 07 A0 5A 50 00 01 02 07
 |...].|./..ZP|
 0010  FF F8 FF F0 C3 40 00 00 00 00 00 00 00 01 6C 28
 |.@l(|
 0020
 matt$ xattr -l /Volumes/Storage/Licensed\ Fonts/Postscript/bradley/
 matt$ xattr -wx com.apple.FinderInfo `xattr -px com.apple.FinderInfo
 Licensed\ Fonts/Postscript/bradley/` /Volumes/Storage/Licensed\
 Fonts/Postscript/bradley/
 matt$ xattr -l /Volumes/Storage/Licensed\ Fonts/Postscript/bradley/
 com.apple.FinderInfo:
   03 99 01 5D 04 7C 02 2F 07 A0 5A 50 00 01 02 07
 |...].|./..ZP|
 0010  FF F8 FF F0 C3 40 00 00 00 00 00 00 00 01 6C 28
 |.@l(|
 0020
 
 After manually setting the extended attribute, the rsync command
 completes without error messages. The process is repeatable, always
 choking on the bradley folder.
 
 After rsync completed without error messages, I compared the sizes of
 source and copy:
 
 matt$ du -k -d1 /Installers/Licensed\ Fonts/
 245196/Installers/Licensed Fonts//Adobe Font Folio - OpenType
 Edition
 51800/Installers/Licensed Fonts//OpenType
 256756/Installers/Licensed Fonts//Postscript
 43308/Installers/Licensed Fonts//TrueType
 598692/Installers/Licensed Fonts/
 matt$ du -k -d1 /Volumes/Storage/Licensed\ Fonts/
 245196/Volumes/Storage/Licensed Fonts//Adobe Font Folio - OpenType
 Edition
 51800/Volumes/Storage/Licensed Fonts//OpenType
 246540/Volumes/Storage/Licensed Fonts//Postscript
 38528/Volumes/Storage/Licensed Fonts//TrueType
 583696/Volumes/Storage/Licensed Fonts/
 
 My understanding is that the sizes should match. The number of items in
 each is the same:
 
 matt$ du -a /Installers/Licensed\ Fonts/ | wc -l
   11916
 matt$ du -a /Volumes/Storage/Licensed\ Fonts/ | wc -l
   11916
 
 I saved the output of 'du -a' for source and copy, then ran a diff.
 There are hundreds of differences, and it appears that many extended
 attributes were not copied. Here's a typical example:
 
 matt$ ls -lh Licensed\ Fonts/TrueType/CCZoinks/CCZoinks.t1 
 -rwx--@ 1 matt  matt 0B Jul 16  1999 Licensed
 Fonts/TrueType/CCZoinks/CCZoinks.t1
 matt$ ls -lh Licensed\
 Fonts/TrueType/CCZoinks/CCZoinks.t1/..namedfork/rsrc
 -rwx--  1 matt  matt   9.5K Jul 16  1999 Licensed
 Fonts/TrueType/CCZoinks/CCZoinks.t1/..namedfork/rsrc
 
 matt$ ls -lh /Volumes/Storage/Licensed\
 Fonts/TrueType/CCZoinks/CCZoinks.t1 
 -rwx--@ 1 matt  matt 0B Jul 16  1999 /Volumes/Storage/Licensed
 Fonts/TrueType/CCZoinks/CCZoinks.t1
 matt$ ls -lh /Volumes/Storage/Licensed\
 Fonts/TrueType/CCZoinks/CCZoinks.t1/..namedfork/rsrc
 -rwx--  1 matt  matt 1B Jul 16  1999 /Volumes/Storage/Licensed
 Fonts/TrueType/CCZoinks/CCZoinks.t1/..namedfork/rsrc
 
 The file is actually a PostScript Type 1 font, where all the content is
 in the resource fork. Rsync created a com.apple.ResourceFork attribute
 on the new file, but the attribute is empty.
 
 After this discovery, I ran my original rsync command again, then
 checked sizes:
 
 matt$ du -kd1 /Volumes/Storage/Licensed\ Fonts/
 245196/Volumes/Storage/Licensed Fonts//Adobe Font Folio - OpenType
 Edition
 51800/Volumes/Storage/Licensed Fonts//OpenType
 254488/Volumes/Storage/Licensed Fonts//Postscript
 42216/Volumes/Storage/Licensed 

Re: Disk image with rsync/rsnapshot

[Henri Shustak]

 I want to make a full disk image backup of my disk with rsnapshot/rsync that 
 I can restore on a new disk.
 Part of my /etc/rsnapshot.conf looks like follows:
 
 exclude   /proc
 exclude   lost+found
 exclude   /media
 exclude   /sys
 exclude   /dev
 exclude   /tmp
 exclude   /dev
 
 
 backup/   localhost/
 
 My questions are ... can I use the backup and copy everything from 
 localhost/hourly.0 --- / of my new disk and is it bootable ?

One thing to keep in mind is that backing up a computer system when it is 
running may cause issues. For example, if you backup the files of a running 
database (eg. MySQL) on a computer system, by backing up the files (which the 
database uses stores information within) by backing up the computers file 
system, then upon a restore of these files, the database may be in an 
inconsistent state. One reason for this is because some data may not have been 
flushed to these files within the file system. However, there other potential 
issues as well.

One option in the above scenario, would be to shutdown the data base during the 
backup. If this is a possibility. Another would be to perform a database dump. 
Then upon a restore to load the dump back back into the database as part of the 
restore process (eg. load the dump back into MySQL). Yet another approach would 
be to have a mirror of the database on a separate system and perform the dump 
on this separate system. There are other approaches as well.

Above is one example of potential issues. For a full bootable system backup. 
You may want to have a look at the Clonezilla project : http://clonezilla.org/

If it is possible to, shutdown the machine and then perform the backup with a 
tool like Clonezilla this will keep your backup and restore process very simple.

Hope this helps.

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Simple Question

[Henri Shustak]

 I have been using rsync on my laptop for some time to safeguard an
 extensive image collection.
 My usage has been: .. rsync -av --delete ~/Pictures /media/disk
 I have found this exceptionally fast ... just as advertised!
 
 I now have set up a new computer system where I want to back up my /home
 to a sata drive.
 My phrase for this is: .. rsync -av --delete /home/david /media/Sata
 Which appears to be identical to my earlier usage .. but .. rsync
 appears to be totally rewriting the backup and instead of being able to
 get a quick refresh, the backup is now taking an hour or more.
 
 Any idea what is going on?


On which operating system are you executing rsync? If you are using Mac OS X, 
often permissions are ignored on non-boot drives by default. 

Information on enabling permissions is available from the following URL 
(Instructions at the following link are specific for Mac OS X systems) : 
http://www.lbackup.org/permissions

This is just one possible reason you are having issues.


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Writing to the wrong directory

[Henri Shustak]

Hello,

Recording the output (standard error and standard out) to a log file will help 
with investigating this issue if it is reproducible. Perhaps another person on 
this list will provide a better suggestion.

In addition, perhaps adding some additional debugging information before and 
after the cron task runs may be helpful. Adding your rsync command into a 
script and then scheduling the script with cron will most likely make adding 
additional debugging information easier.

Hope this helps.

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 3653] Reduce the need for the vanished files warning

[Henri Shustak]

 Doing nightly backups from cron is an *extremely* common use-case of rsync, 
 and
 telling each and every user of rsync that they should either A) write their 
 own
 shell script to filter and discard error messages, or B) play a constant game
 of whack-a-mole chasing down new --exclude options to add, is just silly.  It
 pushes work downstream that could be done upstream much more efficiently.  Why
 make each user write the same code over and over when the developer could 
 write
 it just once, and get it right?

You may want to take a look at LBackup : http://www.lbackup.org
Just one rsync based backup possibility. 

Note : The current version 0.9.8r5-alpha3 of LBackup, requires a wrapper script 
to be
put around rsync if you want to ignore the vanished files warning. : 
http://www.lbackup.org/developer/ignore_vanished_file_warnings

Any assistance to improve rsync / LBackup is warmly welcomed. Perhaps one 
option is 
a new option to suppress this file vanished warning is a possibility? I would 
be interested to have other peoples take on such an approach.

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to mirror only specified directories

[Henri Shustak]

 I'm still struggling to get just the directory(ies) that I want...
 
 You may find this http://tinyurl.com/rsync-exclude-all-include-some post to 
 the LBackup mailing list helpful. 
 
 The example listed (link above) revolves around specifying the root directory 
 as the source and then specifying a selection of sub-directories to include 
 in the sync and then finally a rule which will exclude everything else. 
 
 Please note that each directory within the path must specifically be included 
 for this approach to work. IN addition, this approach utilizes only an 
 excludes file (no include list via the command line).

Following up with the approach listed above, the excludes file quoted below 
should provide the results you are after if modified. Some minor modifications 
should be all that is required.

The example below (extract from URL above) is going to copy the directories 
/private and /home. It will transfer everything recursively within 
/private/etc/, /private/var/ and /home/

 + private 
 + private/etc/***
 + private/var/***
 + home/***
 - *

I understand you wanted to only transfer the /das/htdocs/docs/ directory and 
any contents within this directory recursively with rsync. Provided you are 
syncing the root directory then the following excludes file example listed 
below should make this happen : 

+ das
+ das/htdocs
+ das/htdocs/docs/***
- *

If you have other directories other than /das/htdocs/docs/ to transfer just add 
them in as required.

I think that reiterating the following is worth while : 

 read the man page.  ** and *** can be very useful.

Quoted from the rsync man page : 

 a  trailing  dir_name/*** will match both the directory (as if
 dir_name/ had been specified) and everything in the  directory
 (as  if  dir_name/**  had  been specified).  This behavior was
 added in version 2.6.7


I hope this helps.


-
This email is protected by LBackup, an open source backup solution. 
Free as in freedom; LBackup is licensed under the GNU 
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync as a deliberately slow copy?

[Henri Shustak]

 I'm looking for a way to deliberately copy a large directory tree
 of files somewhat slowly, rather than as fast as the hardware
 will allow. 
 Just do it to localhost - that way it's still a network connection, and
 you can use --bwlimit. Also, you could try nice to lower the
 priority rsync runs at


Yes SSH (depending upon the system) may be resource intensive. As such, the 
suggestion of using nice is a really good suggestion! 

Although, if you are limiting the bandwidth then the resources required by SSH 
will be lower. It will really depend on a number of factors. Bottom line is 
nice is a great suggestion.


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to mirror only specified directories

[Henri Shustak]

 I'm still struggling to get just the directory(ies) that I want...

You may find this http://tinyurl.com/rsync-exclude-all-include-some post to 
the LBackup mailing list helpful. 

The example listed (link above) revolves around specifying the root directory 
as the source and then specifying a selection of sub-directories to include in 
the sync and then finally a rule which will exclude everything else. 

Please note that each directory within the path must specifically be included 
for this approach to work. IN addition, this approach utilizes only an excludes 
file (no include list via the command line).

I learnt about this approach initially from a thread on this (rsync) mailing 
list. I am positive that a similar example will be listed somewhere within the 
the rsync mailing list archives.

Hope this helps.


-
 This email is protected by LBackup, an open source backup solution. 
 Free as in freedom; LBackup is licensed under the GNU 
 http://www.lbackup.org
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Intermittent rsync Issues

[Henri Shustak]

 I posted this to a couple online forums already am am already doubting anyone 
 will be able to directly help me solve my problems.  So I am here to query 
 the experts directly:
 I have a number of rsync clients trying to connect to an rsync server 
 routinely, and they're intermittently failing with one of a couple error 
 messages.
 
 Either:
 
 2010/09/15 13:45:23 [32143] rsync: connection unexpectedly closed (0 bytes 
 received so far) [Receiver] 2010/09/15 13:45:23 [32143] rsync error: 
 unexplained error (code 255) at io.c(601) [Receiver=3.0.7]
 
 Or:
 
 2010/09/15 13:40:01 [7617] rsync: connection unexpectedly closed (0 bytes 
 received so far) [Receiver] 2010/09/15 13:40:01 [7617] rsync error: error in 
 rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7]
 
 The current version of the rsync command I'm using is:
 
 rsync --rsync-path=/usr/bin/rsync --stats --compress --times --links 
 --log-file=/home/ubuntu/rsynclog.txt --exclude thatfile --recursive 
 xxx.xx.xxx.xx:/home/ubuntu/utility_scripts/ /home/ubuntu/utility_scripts 
 
 I previously had --verbose and --progress but removed them after reading on 
 another forum that someone had resolved some latency issues by removing those 
 options. I've also tried this command in the form of a shell script, thinking 
 perhaps the issue was that my rsync client was attempting to reuse an expired 
 ssh connection. To that end, it fails seemingly at random whether using rsh 
 or ssh. It periodically fails whether or not I do --del or --delete, 
 --compress or not, --rsync-path or not.
 
 I cannot get the command to fail from the command line, but when it runs 
 every minute, it fails 5-15 times an hour, depending on the directory being 
 rsync'ed. The permissions and ownership appear to all be correct, and I'm not 
 relying on any sort of environmental variables that would be causing the cron 
 to fail. All of the relevant software packages (bash, rsync, ssh, Linux) are 
 up to date, all key ports are open, and all clients do not fail 
 simultaneously, suggesting this is not a server-side problem.
 
 If anyone has any constructive feedback for resolving or troubleshooting 
 this/these issue/s, I am very interested in hearing it.


Typically how long will the command take to execute?

You could add the 'time' command before rsync to find out.

Hope this helps.


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Backing up a stand alone system to a usb drive with rsync

[Henri Shustak]

 
 I have a stand alone system with two drives / and /home on one and /var
 on the the other. I'd like to backup the complete system to a usb drive.
 I've tried dd however since it copies over everything, even empty space
 it didn't seem very practical since for instance root only has 10GiB out
 of 70 used. Would rsync be a suitable application to use for this or
 will it only work over a network?
 
 See http://edacval.wikidot.com/backup-whole-system-with-rsync


You may also be interested in Clonezilla : http://clonezilla.org/

One possible way of using both of these tools together would be to create a 
full backup using Clonezilla and then use rsync to frequent backups.


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org

The Lucid central Christchurch data center in New Zealand was recently 
affected by a by an earthquake. LBackup allowed services (including 
the LBackup website) to be quickly be restored to new hardware. 




-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Options to copy modified files

[Henri Shustak]

 We use rsync to copy files and directories from one server to the other. What 
 options should I give to rsync so that it only copies the modified files? For 
 example server1 may contain a dir which contains just one file that has been 
 modified, how do I get rsync to copy just that one file over? I would like 
 rsync to copy over the files which have been modified on the server1 to 
 server2.
 
 This is the command I have been using from server2:
 
 /usr/local/bin/rsync -a -A -h -u -p --progress --rsh=/usr/bin/rsh 
 --rsync-path=/usr/local/bin/rsync --exclude=/bin --exclude=/dev 
 --exclude=/etc --exclude=/export --ex
 
 clude=/home --exclude=/lib --exclude=/mnt --exclude=/net --exclude=/opt 
 --exclude=/platform --exclude=/proc --exclude=/sbin --exclude=/system 
 --exclude=/tmp --exclude=
 
 /usr --exclude=/var --exclude=/kernel --exclude=/devices --exclude=patches 
 --exclude=/cdrom r...@191.1.1.85:/ .

By default rsync will copy only the new / modified files. I am guessing that 
you are trying to copy the entire root directory of one server to another, to 
keep these servers in sync. If this is what you are trying to achieve, then you 
must specify the source and destination as arguments to rsync. 

Below are some possible examples (excluding rsync options) : 

# Copy the remote root directory into the current working directory
  $/path/to/rsync r...@191.1.1.85:/ ./

# Copy the current working directory to the remote servers root directory
  $/path/to/rsync ./ r...@191.1.1.85:/

 I would like rsync to copy over the files which have been modified on the 
 server1 to server2.


To achieve this, probably you will want something like the following (again the 
exmple listed below excludes rsync options) :

# Copy the remote root directory to the local root directory.
  $/path/to/rsync r...@191.1.1.85:/ /

If the idea is to have a server ready to swap into place in the event of a 
hardware failure you may wish to look at lsync : 
http://www.lucidsystems.org/tools/lsync

LSync uses rsync to keep servers file systems in sync. The URL above contains a 
diagram explaining how lsync may be used to protect servers in the event of a 
server failure situation.

Setup instructions for a possible server backup are available form : 
http://www.oriontransfer.co.nz/software/lsync/documentation/server-backup/index

Also, you may find that rather than excluding a number of directories and 
files, less setup time is required to simply include the directories you wish 
to keep in sync between the servers.

If you outline in further detail what you are trying to achieve (beyond just 
keeping the servers in sync) then I am hopeful that that others on this list 
will be able to provide further advice which is relevant to what you wish to 
achieve.

I hope this helps. 


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org

The Lucid central Christchurch data center in New Zealand was affected
by a recent earthquake. LBackup allowed services (including the 
LBackup website) to be quickly be restored to new hardware. 


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: OSX 10.6.4 error with -R option

[Henri Shustak]

 I have had reports of problems with the -R option on OSX 10.6.4.
 
 Just tested it myself and found this odd result:
 
 When I run this
 
 dtruss -f  path/to/rsync -aHAXNR --fileflags --force-change 
 --protect-decmpfs --stats -v  /Users/astrid/Documents/main.m  
 /Users/astrid/Desktop/rrr 
 
 it produces the expected results with the relative folder paths in place
 
 /Users/astrid/Desktop/rrr/Users/astrid/Desktop/main.m
 
 but copying file from Desktop itself
 
 dtruss -f  path/to/rsync -aHAXNR --fileflags --force-change 
 --protect-decmpfs --stats -v  /Users/astrid/Desktop/main.m  
 /Users/astrid/Desktop/rrr
 
 /Users/astrid/Desktop/rrr/Users/astrid/
 
 
 Problem solved
 
 Sorry. I didn't even think to check permissions on the Desktop folder, the 
 first logical thing to check. Somehow they had been changed. Fixed them and 
 -R works fine now. I believe other users are likely having permissions issues 
 too.

I have made the same mistake myself and have had the issue reported to me on 
multiple occasions in the past. This is why LBackup now checks the permissions 
are enabled on the destination volume. 

Perhaps when rsync runs on OS X systems it should default to a mode where a 
permissions are enabled check is performed on the destination volume and then a 
warning could be reported if they are not enabled? Just a possibility to avoid 
many more people having this same issue?

Quoted below is an example of the output from LBackup when the permissions on a 
destination drive is not enabled : 

 WARNING! : Permissions are disabled on backup destination volume.
It is recommended that permissions on the destination volume are 
 enabled.
Failure to enable permissions will most likely result in the hard 
 link system failing.
Permissions may be enabled by issuing the following command as 
 root : 
/usr/sbin/vsdbutil -a /Volumes/volume_name

Perhaps adding something like this directly into rsync is a good idea for Mac 
OS X users?

Any comments?

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: compare a directory with itself -- differences found

[Henri Shustak]

 well, when I tell rsync to compare my home directory with itself, it reports 
 many differences.
 
 I'm using rsync  version 3.0.7 protocol version 30.
 
 I was hoping to use it to verify my backup.

What kind of differences are reported. 

Also are you using the the --check-sum option for verification based upon file 
check sums of files rather than the modification times?

-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: compare a directory with itself -- differences found

[Henri Shustak]

I think an important question is what kind of differences do you need to check 
with regards integrity of your copy / backup?

In addition, I suggest that you actually use rsync to copy the data to a 
separate directory and then compare two different directories. The reason I 
suggest this is because it is not clear that you are doing this from your rsync 
command.

The following your rsync commands could be used achieve the results I believe 
you are trying to achieve?

# Make a copy to the /tmp directory :
~/rsync-new-executable --recursive --ignore-times --itemize-changes --xattrs -R 
--exclude-from ~/backup_excludes.txt /Users/pen /tmp/pen

# Check that the copy has completed successfully using checksums and dry run : 
~/rsync-new-executable  --dry-run --checksum --recursive --ignore-times 
--itemize-changes --xattrs -R --exclude-from ~/backup_excludes.txt Users/pen 
/tmp/pen

Another possibility is to use a tool which is designed to compare copies / 
backups on Mac OS X. A great tool which I would highly recommend is 
backup-bouncer : http://www.n8gray.org/code/backup-bouncer/

If you are just after checksums then another possibility is something like 
fingerprint or the various other checksumming / tripwire tools (it really 
depends on the kind of attributes you wish to verify : 
http://www.lucidsystems.org/tools/fingerprint/

I hope this helps.


-
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: DO NOT REPLY [Bug 7629] New: Blank filter file option to ignore whole directory

[Henri Shustak]

 I know that there are external ways to do this, but I thought it would be
 fastest and best as an rsync feature. I've been using rsync for a while now in
 some fairly advanced backup schemes and I'm extremely impressed by it. I was
 thinking it would be quite helpful if there were an option to ignore
 directories with a certain file in them, say .ignore-backup or .ignore-rsync.
 This would make for an easy way to avoid copying excess data, whether it be
 extracted source tarballs, or existing backups.
 
 I know that rsync supports filter files of a certain name, but I'd like an
 option where if it sees the file and it's blank, it simply skips the 
 directory.
 
 So for example:
 
 /home/user/backup/.ignore-rsync exists and is blank, then /home/user/backup is
 ignored, but the it continues onto other folders. If it has filter rules in 
 it,
 it can continue with processing those rules.
 
 Let me know what you think. I believe it would be a very helpful and simple
 feature to add.


An alternative approach would be to run a pre rsync script which builds your 
exclusion list.


-
This message is protected by LBackup
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Scripting rsync against a filesystem with a quota

[Henri Shustak]

 On another note I'm getting an error:
 
 copyfile(.._..v6AxMh,./., COPYFILE_UNPACK) failed:13
 
 When trying to run with -E.
 
 Upstream rsync has no such error message.  If you're using the
 Apple-modified rsync, we don't support it here.


You may wish to compile a version of rsync from source. Below is a link with 
instructions on compiling rsync (and directions on applying some Mac OS X 
patches). 

http://www.lbackup.org/developer/rsync_hfs

To keep things simple you may wish to skip over the steps associated with the 
patches. To do this just skip the two commands quoted below : 

 patch -p1 patches/fileflags.diff
 patch -p1 patches/crtimes.diff


Hope this helps.

--
This message is protected by LBackup
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: HDB, a hard drive backup program

[Henri Shustak]

 Hey all, I'm writing a tool:
 
 http://www.subspacefield.org/security/hdb/



This is an interesting approach. I look forward to seeing the project develop.

Thanks for the link.


--
This email is protected by LBackup
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: backup option

[Henri Shustak]

 I am using rsync to backup files. Is it possible to specify an option
 not to delete files from backup directory if the files are deleted
 from source? In that way, I can always keep something I may or may not
 need in the backup disk, but remove it from a work computer.

As Alex mentioned the omitting the --delete option is the way to go. 

However, you should keep in mind that (depending upon the rsync options you 
provide) if you update a file (within the source directory) the then older 
version of this file in the destination directory will be overwritten. 

If you want to preserve old revisions of files, there are various rsync options 
which will allow various actions to be take place to the older versions which 
already exist in the destination directory.

Below is an example of what I am talking about with regards older versions 
being overwritten in the destination directory when using the --archive rsync 
option.

 $ mkdir 1 2
 $ ls
 1 2
 $ touch 1/hello
 $ rsync -a 1 2
 $ echo hello there  1/hello 
 $ rsync -a 1 2
 $ cd 2/1
 $ ls -a
 . ..hello
 $ cat hello 
 hello there


You will notice that there is no copy of the original (blank) file only the 
more recent version which contains the hello there text.

If you check the rsync manual page and perform some www searches, you will find 
many options and approaches relating to what happens to updated 
files/directories when you run rsync.


--
This email is protected by LBackup
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: IO error causing file deletion failure?

[Henri Shustak]

 readlink_stat(/home/morgan/.gvfs)
 failed: Permission denied (13)

I believe the error above is a problem with the rsync user not being able to 
read the file 
/home/morgan/.gvfs. I suggest that you check the permissions on this file and 
also check that the sudo command is in fact granting root access. 

It may also be worth testing the rsync locally. Initially, this should simplify 
your testing and may reveal the issue as simply a permissions problem.


 IO error encountered -- skipping file deletion


This error suggests that the disk or some other hardware component has a 
problem. However, it may just be the permission error which is causing this 
error to show up.


I am not 100% with either of these errors. I am guessing. However, hopefully 
someone with more knowledge on this list will provide some confirmation or 
additional information to help you further.

With regards to the IO error described you may find the following threads of 
interest:

 - http://www.mail-archive.com/rsync@lists.samba.org/msg02457.html
 (suggests that the IO errors are caused by the permission problem
 This could be true in your case)

 - http://tinyurl.com/lbackup-discussion-io-errors
 (If you find it is not permissions then this thread may be worth a look)



--
This email is protected by LBackup
http://www.lbackup.org






-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Getting rsync to store timing information in its logs

[Henri Shustak]

 Is there a way to know from the rsync logs how long it took to do a backup?
 
 The only timing info,. I see is this at the end:
 
 sent 3067328 bytes  received 7853035429 bytes  1187888.83 bytes/sec
 total size is 1559866450336  speedup is 198.55


The latest alpha release of LBackup supports new configuration options allowing 
the time required for various backup related operations to be reported.

One such option is enable human readable reporting on the timing of the 
snapshot generation. Quoted below is an example of the output from the latest 
alpha release. 

 total size is 106717731958  speedup is 831.04
 Time required for snapshot generation : 0 days, 0 hours, 20 minutes, 17 
 seconds

The current granularity of the timing system mentioned above is 1 second. 
LBackup is using the the 'date' command to calculate timing.

If you are using the shell to measure the time required and you require 
increased granularity with simplicity, then I would recommend the use of the 
'time' command.

A majority of this post is from the following LBackup discussion thread : 
http://tinyurl.com/lbackup-discussion-human-read

Finally, I believe that 118788 bytes/sec is approximately 1.2 MB/sec

Hope this helps.


--
This email is protected by LBackup
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync algorithm

[Henri Shustak]

 Check file size makes sense, but how rsync checks times? If a file is
 copied from one side to another remote side, the time will be
 different, right?
 
 Sorry wrong question, the copy file should be able to reserve the mtime.

If the rsync --times option is used then rsync will attempt to preserve 
modification times. 

You version of rsync may also have support of the --crtimes option. If this 
option is used then rsync will attempt to preserve create times.


--
This email is protected by LBackup
http://www.lbackup.org




-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to block this fiel name from being transfered via rsync?

[Henri Shustak]

 Are you specifying this lines in an excludes file?
 
 yes

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync sequence

[Henri Shustak]

 Can I just list them one after the other like so in crontab -e

You may wish to also consider using a single rsync command :

rsync -auv --delete \
/Godfather/Documents \
/Godfather/Setups \
/Godfather/Pictures \
/Godfather/Backups \
/Godfather/Videos \
/Backup # backup destination

With this approach I am not sure which order the source directories will be 
copied? If the execution order is critical then you are probably best to stick 
to a separate command for each source directory in a script as has been 
previously suggested.

Finally, I recommend excessing extreme caution when using the --delete option.
  
I hope this helps.


---
This email is protected by LBackup, 
an open source backup solution.
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --compare-dest weirdness

[Henri Shustak]

 the -i shows that most are being copied due to time differences, so in
 theory -t should work? This does in fact work on a little test setup
 on my work laptop, i will test it properly when i get home tonight.



I am glad that the -t option is working with your test.

If you decide to use LBackup and have a full hard linked backup then a possible 
option is to have a post action script which passes a list of files and 
directories which have been updated or created to tar for archive creation. 

To enable rsync session logging add the following line to the LBackup backup 
configuration file : 
 
 enable_rsync_session_log=YES

Then using tar with the -n option you would be able to feed in a list of newly 
created or modified files.

Depending upon what kind of information you want to preserve something like the 
following could do the trick. However, please keep in mind that you may need to 
add or remove tar various options.

 tar -c -n -z -v -f /tmp/new_archive.tar.gz -T /tmp/list_to_archive.txt 

If you opted for such an approach you could potentially stop using the 
--compare-dest option and rely upon the output from rsync for a list of files 
you would like to archive.

All the best with your backup script. 

With any luck the rsync -t option will work with you backup as well as it did 
for your testing.


---
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync backup issues using relative paths and LVM snapshots

[Henri Shustak]

 I did think about remotely executing a mkdir before the backup, but one 
 blocker is that I will be using Thecus NAS boxes as some off-site locations 
 and I don't have shell access.
 
 You could mkdir the directory locally somewhere (anywhere), and rsync
 just that directory to the remote side, affectively doing a remote mkdir
 via rsync.  Repeat for deeper directories. If the remote directory
 already exist, it shouldn't do any harm.



Great thinking! That would certainly work. Such an elegant approach!

This would even work if you want to create a hierarchy you just need to use the 
rsync --recursive option.


-
This email is protected by LBackup, an open source backup solution. 
Free as in freedom; LBackup is licensed under the GNU GPL
http://www.lbackup.org

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --compare-dest weirdness

[Henri Shustak]

 interestingly, i tried to see if something was wrong with my statments by 
 doing:
 mkdir ~/rsynctest/dir1
 mkdir ~/rsynctest/dir2
 mkdir ~/rsynctest/dir3
 
 nano ~/rsynctest/dir1/file1 (wrote the line hello world and saved)
 nano ~/rsynctest/dir1/file2 (wrote the line hello and saved)
 
 cp ~/rsynctest/dir1/file* ~/rsynctest/dir2/
 
 checked md5sums of both files in both dirs to ensure they were identical
 
 the did rsync -rvu --compare-dest=/home/kevin/rsynctest/dir2/
 /home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/
 
 the two files from dir1 were copied to dir3, even though identical
 copies, with the same names, existed in dir2.
 
 i also tried this again with the compare dir relative to the dest dir,
 i.e.: rsync -rvu --compare-dest=../dir2/ /home/kevin/rsynctest/dir1/
 /home/kevin/rsynctest/dir3/ again with same result



I thought about this further and I think the problem you are having is that the 
modification times are not being preserved and you are not using the --checksum 
option. 

As such, I suggest that you add the --times or --checksum option and see if 
that helps. 

I suspect that this will solve your problem. Let me know =:^)


-
This email is protected by LBackup, an open source backup solution. 
Free as in freedom; LBackup is licensed under the GNU GPL
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync backup issues using relative paths and LVM snapshots

[Henri Shustak]

 I really want to put the logic in the script so it is easy to bring another 
 backup location online easily.

If you have shell access to the destination system from your backup script then 
one option may be to issue 'mkdir -p' via ssh.

 Creating the directories manually on the destination server is not desirable

If you using mkdir command via ssh is what you are referring to as a manual 
process then okay another approach will be required.

If your OS has support for sparse bundle images then one possible approach 
could be staging the backup to local media and then once finished unmount the 
sparse bundle image and then sync this image your remote locations. Below is a 
link to a script (which currently only supporting Mac OS X) which will 
synchronize a sparse bundle image to a remote server.

I would also suggest that a pull backup strategy may also be worth considering. 

  - http://www.lbackup.org/network_backup_strategies

If you are distributing your backups to multiple systems one advantage of a 
pull backup strategy is the ability to move the job of distribution to the 
backup destination system.



--
This email is protected by LBackup
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync backup issues using relative paths and LVM snapshots

[Henri Shustak]

 Below is a link to a script (which currently only supporting Mac OS X) which 
 will synchronize a sparse bundle image to a remote server.

Sorry I forgot the link in the previous email. 

  - http://www.lbackup.org/synchronizing_disk_images_between_machines

Finally, yes I would be very interested to know how you get on with this 
project. As such, please keep me (and the list) posted.


--
This email is protected by LBackup
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --recursive and -H

[Henri Shustak]

 Aaah, thanks for the insight.  RSYNC must transfer the file regardless of the 
 link count, but it also takes note of the missing link.  So, it probably 
 considers an inode with multiple links resolved only after it finds a 
 brother/sister link and deletes the duplicate.

You may also be interested in the rsync --link-dest option. The rsync 
--link-dest option is used in different ways by a variety of backup scripts and 
utilities.

A screen cast about how this works with LBackup is available from the URL below 
: 
 
  - URL http://www.lbackup.org/screencasts 
  - Title Basic Backup Local Machine


Finally, you may find the following list of commands and output of interest : 

  # touch /tmp/file1
  # ls -l /tmp/file1 
  -rw-r--r--  1 root  wheel  0 Jun 29 07:34 /tmp/file1
  # ln /tmp/file1 /tmp/file2
  # ls -l /tmp/file1 /tmp/file2
  -rw-r--r--  2 root  wheel  0 Jun 29 07:34 /tmp/file1
  -rw-r--r--  2 root  wheel  0 Jun 29 07:34 /tmp/file2


Hope this helps.


 This email is protected by LBackup
 http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: where is backup-dir rooted?

[Henri Shustak]

 [What I am trying to achieve is to backup (part of) my home directory to a 
 backup server such that
 
 a) I have a reasonably up to date (within a day if I do it overnight) of my 
 current state
 b) If I have deleted or updated a file the old version of it gets placed into 
 a special snapshot directory away from this backup.



I am positive that others on this list will be able to assist you further with 
this question. 

In the mean time you may want to refer to a recently started thread relating to 
backups on the rsync mailing list : 
http://www.mail-archive.com/rsync@lists.samba.org/msg25924.html

The summary of my reply to this thread was that you may want to have a look at 
LBackup.

I hope this helps.



--
This email is protected by LBackup
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: few questions on rsync

[Henri Shustak]

 Reversing --backup logic:
 Currently, if --backup is used (and --backup-dir), a copy of existing
 file that is replaced on rsync will be placed there. Is there a way to
 keep original copy (ie. base) the same, but just place whatever has
 changed to a different location? Taking backup to a central backup
 server with 7 day incremental example on rsync page, but reverse the
 logic. Instead of current, you have 'initial' and then each
 incremental only contains changed files from 'initial' (basically,
 each incremental would then be differential, but thats besides the
 point).

One thing to keep in mind is that when performing a restore, you may want to 
recreate a directory structure from the backup rather than just restore a 
single file.

As such, LBackup http://www.lbackup.org may be worth considering. However, 
you may not need the ability to recreated the directory structure from your 
backup and as such it may not be the right tool for you.

LBackup uses hard links to store each backup snapshot as a complete backup 
tree. This snap shot approach allows you to browse your backup tree. When you 
perform a restore from the backup tree you are able to restore individual 
files. However, you are also able to recursively restore a directory structure. 
In addition, you are able to utilize additional utilities (eg. find, grep, 
etc...) to assist with the restore of specific files or entire directory 
structures generated on the fly with scripts.

In addition, LBackup includes the lcd command 
http://www.lbackup.org/command_line_tools which allows you to change your 
current working directory forwards and backwards though time. This assists with 
locating the files and directories which you wish to restore. 

The lcd command bundled with LBackup is scriptable and that means that you have 
the ability to have create scripted backup tree searches. To see lcd in action 
visit the URL below and click on the link to the lcd screen cast : 
http://www.lbackup.org/screencasts

 (stream changes into single file for example, put it to tape etc.)

LBackup may not be an option for you as it has no support for backup directly 
to tape. LBackup used rsync extensively for the coping of files. As I 
understand the way which rsync (currently) works there is no support for rsync 
to sync directly to tape.

You should be able to use tar after running rsync to disk, in order to transfer 
your files to tape. Perhaps LBackup will assist you with your D2D backups? 

More backups the better! 



--
This email is protected by LBackup
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Simple whole volume copy

[Henri Shustak]

 Sorry, I'm only going to be doing this a single time. I don't really need a 
 script, just a simple command 


I urge to to read this great write up Cloning Mac OS X disks by Mike Bombich 
: http://www.bombich.com/mactips/image.html

If you have access to a test computer system running Mac OS 10.6, then I 
recommend that you familiarize yourself with rsync on that separate test 
system. If you do not have a separate test system available, then plug in a 
memory stick or create a disk image and do some testing prior to running this 
on your destination drive. 

I find the following rsync options '--dry-run', '--verbose' and 
'--log-file=FILE' options are useful for testing purposes.

Also, would you kindly answer the following questions for clarification : 

(1) Do you have a destination volume with some of the files from the source 
volume already present? 

(2) You want to make the destination volume exactly the same as the source 
volume even if that means some files will be deleted on the destination volume?

(3) Assuming you already have some files on the destination volume how were 
they initially copied to this destination volume?

 which will preserve all the flags.

It is a good idea to be clear about which meta data is being preserved and how 
the copy is going to take place. In addition, it is also a good idea to check 
that the copy has preserved the meta data you are expecting. 

For example, if it is important that you preserve hard links from the source 
volume on the destination volume then it is imperative that you use the 
'--hard-links' rsync option. Also, if you use the '--hard-links' option then it 
is important that the destination volume has a file system which will support 
hard links. Once the copy has completed you should then check that the hard 
links have been copied across as you expected.

Another example is symbolic links and how they are to preserved. There are 
various rsync options which will change the way symbolic links are handled. 
Quoted below is a snippet from the rsync man page.

  -l, --links copy symlinks as symlinks
  -L, --copy-linkstransform symlink into referent file/dir
  --copy-unsafe-links only unsafe symlinks are transformed
  --safe-linksignore symlinks that point outside the tree
  -k, --copy-dirlinks transform symlink to dir into referent dir
  -K, --keep-dirlinks treat symlinked dir on receiver as dir


If you are still unsure of how to proceed then please let me know.



--
This email is protected by LBackup
http://www.lbackup.org



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Simple whole volume copy

[Henri Shustak]

 Sorry, I'm only going to be doing this a single time. I don't really need a 
 script, just a simple command which will preserve all the flags.

Also, another question, what kind of data is on this volume that you are 
backing up? 



--
This email is protected by LBackup
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Simple whole volume copy

[Henri Shustak]

 I have a volume at 192.168.0.2 on my local network. I'd like to rsync the
 entire volume to a backup volume, skip all files already present on the
 backup volume (same name and the same or earlier timestamp)


Firstly, I would recommend that on both machines you compile and install a copy 
of the latest version of rsync with some Mac OS X specific patches the link 
below will assist you : 
http://www.lbackup.org/developer/rsync_hfs

If you are not coping a boot volume, then you will probably find the 
information at the following URL helpful : 
http://www.lbackup.org/synchronizing_disk_images_between_machines

If you are trying to create a bootable clone of a startup disk then I would 
recommend Carbon Copy Cloner (I would also recommend that you shutdown the 
system before you make a clone) : http://www.bombich.com/

Whichever method you use, I recommend the use of Backup-Bouncer to check that 
it has actually copied the various meta-data you are expecting to have 
preserved. Further information regarding backup bouncer is available from the 
following URL : http://www.n8gray.org/code/backup-bouncer/

 include all flags, dates, etc. as applicable to Mac OS X 10.6.x.


I am unsure if the latest version of Backup-Bouncer or rsync supports the 
checking / coping of HFS file system compressed files. Hopefully someone else 
with a deeper understanding of this matter is watching this thread and will be 
able to comment.

Further information regarding the file system compression on Mac OS 10.6 is 
available at the following URL : 
http://www.macosxhints.com/article.php?story=20090902223042255

There is a recent thread on the rsync mailing list which relates to the rsync 
flag : 

  --protect-decmpfs

The full thread is available from the following URL : 
http://www.mail-archive.com/rsync@lists.samba.org/msg25797.html

Finally, to be absolutely sure that all meta data is preserved you could 
consider a tool such as dd to copy the disk block by block. However, using a 
tool such as dd will mean that you will miss out on the benefits of a tool like 
rsync : http://linux.die.net/man/1/dd




--
This email is protected by LBackup
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Simple whole volume copy

[Henri Shustak]

 same name and the same or earlier timestamp

Would you please provide some clarification on your desired result in the 
following situation. 

file_a : /Volumes/src/file.txt
file_b : /Volumes/dst/file.txt

If file_b has a newer modification time than file_a will you want file_a to 
overweight file_b?

Thank you for this clarification.


--
This email is protected by LBackup
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Simple whole volume copy

[Henri Shustak]

 Right. That's why I need rsync. I'm really not looking for a utility, but a
 command line program which will do incremental backup, and rsync is the only
 one I'm aware of that does that in Mac OS.

There are also some other options available. Although, I have not tested them 
on Mac OS X 10.6. 

  - PsyncX is one project which jumps to mind : 
http://psyncx.sourceforge.net/.

  - Even more options are listed at the following URL : 
http://www.lbackup.org/alternatives

If you are after incremental backup then you may be interested in the 
'--backup' and 'backup-dir' arguments. In addition, the '--link-dest' option of 
rsync may be useful. This option allows for incremental hard linked backups as 
outlined by Mike Rubel : http://www.mikerubel.org/computers/rsync_snapshots/

LBackup makes use of Mike Rubel's hard linked approach for backup sets : 

  - LBackup home page : 
http://www.lbackup.org

  - Screen casts on how LBackup works : 
http://www.lbackup.org/screencasts

  Note : I recommend having a look at the Basic Backup Local Machine
 screen cast to determine if LBackup is an option worth further
 consideration.

If you are going to roll your own backup script around rsync then as previously 
mentioned, the following URL will probably be very useful with regards the 
flags you may wish to use to preserve various meta data on Mac OS X : 
http://www.lbackup.org/synchronizing_disk_images_between_machines
 
Quoted below is an extract from the script (available from the above link). 
This command has many flags which will be of interest to you for an incremental 
copy which preserves Mac OS X meta-data. However, as previously mentioned I 
recommend that you use Backup-Bouncer or some other approach to check that 
rsync is preserving what you require.

   # Use rsync to copy / update the remote file
 
   ${path_to_rsync} --rsync-path=${path_to_rsync} \
   -aNHAXEx --delete --progress --stats --protect-args --fileflags 
 --force-change \ 
   ${local_source_directory} \ 
   
 ${remote_server_us...@${remote_server_address}:${remote_destination_directory}

Before running rsync any of the options listed above I suggest that you check 
the rsync man page / rsync patch documentation for details. In particular, be 
very carful if you use the '--delete' option with rsync. Also, note the command 
quoted above makes no mention of the '--link-dest' option. 

The LINUX man page for rsync is available at the following URLs (just to name a 
couple) :

  - http://linux.die.net/man/1/rsync
  - http://www.samba.org/ftp/rsync/rsync.html


Personally, I like rsync very much. It is a great tool with an amazing 
community!


--
This email is protected by LBackup
http://www.lbackup.org


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html