Re: Skipping hardlinks in a copy
On Wed, Mar 07, 2007 at 09:22:08PM -0800, Sriram Ramkrishna wrote: Hi folks, I've been googling around for awhile but I can't seem to find an answer to my question. I have a number of filesystems that contain thousands of hard links due to some bad organization of data. Rsync, cpio and various other utilities fail to copy this data because I think there might be some cycles in it. (you know you have troubles if cpio can't copy it!) What I thought I would do instead is to copy the data but skip any files that are hard links. Then after the copy is finished, I will use some kind of find . -type l type command that finds the hard links and then make a script to recreate it. This saves me a lot of trouble with not having to stat the files and not having the receive side balloon up. Is there a way to have it skip hard links when doing an rsync? Or is there some other mystic incantation that I can use that might accomplish the same thing. Surely a hard link is just 'a file', that's what a file is, thus it's impossible to skip them without skipping everything (except symbolic links, FIFOs, etc). The only clue to something having more than one link to it is the 'number of links', but then how do you decide which link is the 'right' one to copy as they're all the file. -- Chris Green -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Skipping hardlinks in a copy
On Wed 07 Mar 2007, Sriram Ramkrishna wrote: that are hard links. Then after the copy is finished, I will use some kind of find . -type l type command that finds the hard links and then find -type l will find symbolic links, *not* hard links. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
about lock.
Hi, i try to learn how rsync lock files for create a backup, but.. not find any on www. some can help me and give any HOWTO or FAQ or document ? Thanks! -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: about lock.
On Thu 08 Mar 2007, Alejandro Feij?o wrote: Hi, i try to learn how rsync lock files for create a backup, but.. not find any on www. rsync does not lock files. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: about lock.
mmm and is posible corrupt data??? for example if rsync is on file called A.txt and at same time mysql is read A.txt... whats happend? is that who i dont understand. 2007/3/8, Paul Slootman [EMAIL PROTECTED]: On Thu 08 Mar 2007, Alejandro Feij?o wrote: Hi, i try to learn how rsync lock files for create a backup, but.. not find any on www. rsync does not lock files. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Skipping hardlinks in a copy
On Wed, Mar 07, 2007 at 09:22:08PM -0800, Sriram Ramkrishna wrote:
| Hi folks, I've been googling around for awhile but I can't seem to find
| an answer to my question.
|
| I have a number of filesystems that contain thousands of hard links due
| to some bad organization of data. Rsync, cpio and various other
| utilities fail to copy this data because I think there might be some
| cycles in it. (you know you have troubles if cpio can't copy it!)
|
| What I thought I would do instead is to copy the data but skip any files
| that are hard links. Then after the copy is finished, I will use some
| kind of find . -type l type command that finds the hard links and then
| make a script to recreate it. This saves me a lot of trouble with not
| having to stat the files and not having the receive side balloon up.
|
| Is there a way to have it skip hard links when doing an rsync?
| Or is there some other mystic incantation that I can use that might
| accomplish the same thing.
The following command pipeline can give you a list which you could isolate
to being just the first ocurrence of each file that is sharing the same
inode:
find . ! -type d -printf '%10i %P\n' | awk '{n=substr($0,12);if(a[$1]==1){print
other,n;}else{a[$1]=1;print first,n;}}'
Note the above is 123 characters long. You may have issues with mail
programs that truncate or wrap it around, so be careful. The fixed
size formatting of the inode number in the find output is to make it
easy to extract the name, or the name plust the symlink target, in the
awk command using substr().
One approach in the situation you have, if the filesystem is not corrupt
(which it might be, because files don't create cycles), is to create a
list of files based on their inode number, and hardlink each file to one
named by its inode number. Just rsync the directory full of inode numbers.
Then re-expand on the destination based on that list.
You should not be following symlinks in a file tree recursion. Rsync,
find, cpio, and others, know not to.
But I suspect some kind of filesystem corruption, or at least some hard
links being applied to directories. The latter can create cycles if not
done carefully (and there is virtually no case to ever do that at all by
intent).
I do not consider it bad organization to have lots of files be hardlinked.
In fact, I have a program that actually seeks out indentical files and makes
them be hardlinked to save space (not safe in all cases, but safe in most).
The command find . -type l will only find symlinks. You can find files
that have hard links with find . ! -type d -links +1 -print. Note that
all file types can have hard links, even symlinks. Do exclude directories
as those will have many links for other reasons (e.g. 1 for self reference,
1 for being inside a directory and 1 each for each subdirectory within).
--
|---/--|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / [EMAIL PROTECTED] |
|/-|
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: about lock.
On Thu, Mar 08, 2007 at 12:08:55PM +0100, Alejandro Feij?o wrote: | mmm and is posible corrupt data??? | | for example if rsync is on file called A.txt and at same time mysql is read | A.txt... whats happend? is that who i dont understand. It can get mixed data. The file contents could appear corrupt to another mysql running on the target. You can use rsync to backup database files while mysql is running to get _MOST_ of the content transferred while the database is running. Repeat rsync until very little change happens in a shorter run. Then shut down mysql cleanly and do a final rsync to sync the files in a non-corrupt way. At least mysql downtime is minimal this way. You may also want to employ methods of backup up data from within mysql itself on a row/column basis as transactions update things, to a live backup mysql engine running elsewhere that can take over serving clients should the first die. This has the advantage of a hot failover AND you could use that backup engine to do the rsync backups from, since only it needs to be shutdown to do the final rsync syncronization. Just be sure the in-mysql backups know how to re-syncronize everything that was changed while the backup mysql was briefly down for the last rsync run. /-- rsync copy to site 2 main mysql == backup mysql -- rsync copy to site 3 \-- rsync copy to site 4 -- |---/--| | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below | | first name lower case at ipal.net / [EMAIL PROTECTED] | |/-| -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: about lock.
Its Great! thanks for the info :) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[BUG] clear ACL-s on destination
Destroy ACL-s on destination when no ACL-s differens between source and destination. Bug is somehow related with function send_file_name() called with negative file descriptor f. There is no such bug in 2.6.9 version, but there options -X -A --deleted can't be used (we have Internal error: wrong write used in receiver.). If I fix this, avoid calling send_acl() send_xattr() with negative descriptor I get the same ACL bug in 2.6.9 . -- Stanislaw Gruszka Sprawdź ile jest wart Twój samochód! W ycena aut jak na dłoni - http://klik.wp.pl/?adr=www.wycenyaut.wp.plsid=1047 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Receive more information about transfer problems in server log?
Hi Dave, Dave Markham wrote: What are you the Rsync options you are using ? Clientside: rsync -vratz --password-file=somefile --force --delete-excluded --delete --exclude-from=somefile --files-from=somefile Serverside: rsync --no-detach --daemon --config /etc/rsyncd.conf with rsyncd.conf: motd file = /etc/rsyncd.motd log file = /var/log/rsync/rsyncd.log pid file = /var/run/rsyncd.pid secrets file = /etc/rsyncd.secrets read only = no uid = rsync use chroot = yes transfer logging = yes timeout = 14400 max connections = 25 incoming chmod = u+rw Thanks for your help! Regards, -- Bas On 2/23/07, *Bas van Schaik* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Hi all, I'm probably using the most basic rsync-setup: just running the rsync-daemon and use plain transfers (no encryption or other shells, plain rsync). Transfer logging is enabled. I've got a lot of client machines rsyncing their backups, both Linux and Windows. For some rsyncs, the log tells me this: 2007/02/23 02:42:04 [29077] connect from ***.nl (*.*.*.*) 2007/02/23 02:42:05 [29077] rsync to 192168162 from [EMAIL PROTECTED] (*.*.*.*) 2007/02/23 02:42:05 [29077] receiving file list 2007/02/23 02:42:06 [29077] IO error encountered -- skipping file deletion 2007/02/23 02:42:06 [29077] rsync / 2007/02/23 02:42:06 [29077] rsync error: some files could not be transferred (code 23) at main.c(872) [generator=2.6.9] (all entries about PID 29077) I can see that something went wrong during this sync. But what exactly? Which files could not be transferred exactly, and why? According to the error code (23), some files where partially transferred due to an error, but there are no files in the transfer log! And which error did occur exactly? Of course, I can browse the client logfiles every day, but that isn't really an option: there are more than 50 clients! Can anyone help me getting more information in my server log about transfer problems? Thanks! Regards, -- Bas van Schaik -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete --force Won't Remove Directories With Dotnames
Hi Matt, On 2 Mar 2007 at 10:37, Matt McCutchen [EMAIL PROTECTED] said: On 3/2/07, Sabahattin Gucukoglu [EMAIL PROTECTED] wrote: --delete --force Won't Remove Directories With Dotnames if you remove a directory which was aborted in a recent transfer on source (leaving a .partial mirror at dest) the .partial directory won't go away on next rsync, ever. Example: rsync --partial-dir=.partial /foo /bar. Abort the transfer, and /foo/biguns/bigfile gets left in /bar/foo/biguns/.partial/bigfile. Now remove /foo/biguns and do the transfer again, to completion. The /bar/foo/biguns/.partial/bigfile is still there. Rsync is leaving .partial around because it's a partial dir, not because its name begins with a dot. Right, yep, I've tested this and you're right. However, rsync is still never making any mention of the .partial directories in the output during the list build or sync (even with -vv, and I think even this would be a courtesy indicator). As the explanation of --partial-dir in the man page says, to get rsync to delete old partial dirs, supply your own rule --filter='R .partial/' and use --delete-after to avoid deleting partial dirs until after rsync has gotten what it needs from them. Thanks! I *honestly* never saw this in the manual before. Maybe it was never there (I started out with rsync 2.6.6). This answer works but to me still seems ... not elegant. Maybe rsync should automatically delete a directory's partial dir when it finishes processing files immediately in that directory. What possible reason does rsync have for leaving them around? Not to do so means excluding a directory from deletion even when this is merely an aid to implementation, rather than the function of rsync with --delete --force. I won't yet update rsync as this is packaged and isn't easy to install cleanly if patched (this is Gentoo, for which rsync is a rather necessary piece of software :-) ) but if I get a chance I'll have a look at the new implicit rule stuff and see what I can make of it. For now I can just search for the .partial directories. Thanks for your help. Cheers, Sabahattin -- Sabahattin Gucukoglu mailatsabahattindashgucukogludotcom Address harvesters, snag this: [EMAIL PROTECTED] Phone: +44 20 88008915 Mobile: +44 7986 053399 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Tip for SSH users with connection unexpectedly closed troubles
Thanks for rsync - it's great! I use it for all my backups. I use rsync via SSH. Recently - I was having trouble with my backups: rsync: connection unexpectedly closed (4968349 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(453) [receiver=2.6.9] After following all the steps in the FAQ and issues.html on the rsync website, searching the mailing list, upgrading rsync, using the rsync-debug script, and consulting a number of people someone suggested that I add this to my ~/.ssh/config file: ServerAliveInterval 5 ServerAliveCountMax 120 This fixed the problem -- apparently something between my backup and primary servers was cutting the connection during pauses (I'm still not sure which was, even after research all the points of failure I could). I don't know if many other people have this trouble. But if it is a common problem, it'd be great to add a suggestion to issues.html under Q: Why does my transfer die with something like the following error? If you are using rsync over SSH, make sure the connection isn't being closed due to inactivity. You may need to add ServerAliveInterval and ServerAliveCountMax parameters to your client side ~/.ssh/config file. Thanks! Phil -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete --force Won't Remove Directories With Dotnames
On Thu, Mar 08, 2007 at 05:02:21PM -, Sabahattin Gucukoglu wrote: However, rsync is still never making any mention of the .partial directories in the output during the list build or sync (even with -vv, and I think even this would be a courtesy indicator). Starting with version 3.0.0, rsync will mention any directory that it would like to delete, but can't due to exclude/protect restrictions (thanks to Matt's nice work). So, this will improve in the next release. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Skipping hardlinks in a copy
On Wed, Mar 07, 2007 at 09:22:08PM -0800, Sriram Ramkrishna wrote: Is there a way to have it skip hard links when doing an rsync? If you mean you want to skip any file that has more than one link, you could do this: find . -type f -links +1 /path/exclude.txt Then, you'd use the exclude.txt file via the --exclude-from option. However, you mentioned loops, and that makes me think that the problem is not with file loops, but dirs looping back in the hierarchy. The above won't help you if that is the case (since find will loop too). ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [BUG] clear ACL-s on destination
On Thu, Mar 08, 2007 at 03:00:38PM +0100, Stanis?aw Gruszka wrote: Destroy ACL-s on destination when no ACL-s differens between source and destination. Can you explain this a little more? we have Internal error: wrong write used in receiver. Yes, this is a known problem with the acls.patch and xattrs.patch in 2.6.9: http://lists.samba.org/archive/rsync/2006-November/016706.html http://lists.samba.org/archive/rsync/2006-November/016710.html ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Tip for SSH users with connection unexpectedly closed troubles
On Thu, Mar 08, 2007 at 10:32:04AM -0700, Phil Hassey wrote: someone suggested that I add this to my ~/.ssh/config file: ServerAliveInterval 5 ServerAliveCountMax 120 Those options only apply to protocol 2 connections (which people should be using anyway these days). There are also settings for KeepAlive, ClientAliveInterval, and ClientAliveCountMax that someone might want to fiddle with. it'd be great to add a suggestion to issues.html under Q: Why does my transfer die with something like the following error? Good idea. Thanks! ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--link-dest on a local filesystem
Hi guys, I've been struggling with getting --link-dest working for a couple of hours now. I'm using rsync 2.6.9 (protocol 29), on Gentoo Linux. For some strange reason, whenever I recreate the source file, even though it's identical to the old source file --link-dest simply does not create the link. Point in case (starting from a blank directory tmp in my home directory): [EMAIL PROTECTED] ~/tmp $ mkdir a [EMAIL PROTECTED] ~/tmp $ echo foo a/tmp [EMAIL PROTECTED] ~/tmp $ rsync -va --link-dest=../b.1 a/ b.0/ building file list ... done created directory b.0 ./ tmp sent 142 bytes received 48 bytes 380.00 bytes/sec total size is 4 speedup is 0.02 [EMAIL PROTECTED] ~/tmp $ rm a/tmp [EMAIL PROTECTED] ~/tmp $ echo foo a/tmp [EMAIL PROTECTED] ~/tmp $ mv b.0 b.1 [EMAIL PROTECTED] ~/tmp $ rsync -va --link-dest=../b.1 a/ b.0/ building file list ... done created directory b.0 ./ tmp sent 143 bytes received 49 bytes 384.00 bytes/sec total size is 4 speedup is 0.02 [EMAIL PROTECTED] ~/tmp $ md5sum b.*/tmp d3b07384d113edec49eaa6238ad5ff00 b.0/tmp d3b07384d113edec49eaa6238ad5ff00 b.1/tmp [EMAIL PROTECTED] ~/tmp $ stat b.*/tmp File: `b.0/tmp' Size: 4 Blocks: 8 IO Block: 4096 regular file Device: 306h/774d Inode: 183267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1001/ jkroon) Gid: ( 100/ users) Access: 2007-03-08 20:50:31.0 +0200 Modify: 2007-03-08 20:50:24.0 +0200 Change: 2007-03-08 20:50:31.0 +0200 File: `b.1/tmp' Size: 4 Blocks: 8 IO Block: 4096 regular file Device: 306h/774d Inode: 183111 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1001/ jkroon) Gid: ( 100/ users) Access: 2007-03-08 20:50:12.0 +0200 Modify: 2007-03-08 20:49:51.0 +0200 Change: 2007-03-08 20:50:12.0 +0200 Ok, so perhaps the -a which includes --times mucks things up, so add --no-times: [EMAIL PROTECTED] ~/tmp $ rm -rf * [EMAIL PROTECTED] ~/tmp $ mkdir a [EMAIL PROTECTED] ~/tmp $ echo foo a/tmp [EMAIL PROTECTED] ~/tmp $ rsync -va --no-times --link-dest=../b.1 a/ b.0/ building file list ... done created directory b.0 ./ tmp sent 142 bytes received 48 bytes 380.00 bytes/sec total size is 4 speedup is 0.02 [EMAIL PROTECTED] ~/tmp $ rm a/tmp [EMAIL PROTECTED] ~/tmp $ echo foo a/tmp [EMAIL PROTECTED] ~/tmp $ mv b.0 b.1 [EMAIL PROTECTED] ~/tmp $ rsync -va --no-times --link-dest=../b.1 a/ b.0/ building file list ... done created directory b.0 ./ tmp sent 143 bytes received 49 bytes 384.00 bytes/sec total size is 4 speedup is 0.02 [EMAIL PROTECTED] ~/tmp $ md5sum b.*/tmp d3b07384d113edec49eaa6238ad5ff00 b.0/tmp d3b07384d113edec49eaa6238ad5ff00 b.1/tmp [EMAIL PROTECTED] ~/tmp $ stat b.*/tmp File: `b.0/tmp' Size: 4 Blocks: 8 IO Block: 4096 regular file Device: 306h/774d Inode: 183267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1001/ jkroon) Gid: ( 100/ users) Access: 2007-03-08 20:53:03.0 +0200 Modify: 2007-03-08 20:53:03.0 +0200 Change: 2007-03-08 20:53:03.0 +0200 File: `b.1/tmp' Size: 4 Blocks: 8 IO Block: 4096 regular file Device: 306h/774d Inode: 183111 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1001/ jkroon) Gid: ( 100/ users) Access: 2007-03-08 20:52:51.0 +0200 Modify: 2007-03-08 20:52:51.0 +0200 Change: 2007-03-08 20:52:51.0 +0200 [EMAIL PROTECTED] ~/tmp $ Still no hard link ... If someone could just explain to me why this behaviour happens, and if it can be avoided, it would be much appreciated. Jaco -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Skipping hardlinks in a copy
On Wed, Mar 07, 2007 at 09:22:08PM -0800, Sriram Ramkrishna wrote:
Hi there,
For some reason, I sent this mail before I was fully subscribed and I
have missed out on the replies. If I don't answer all the responses this
is why.
The following command pipeline can give you a list which you could
isolate to being just the first ocurrence of each file that is sharing the
same inode:
find . ! -type d -printf '%10i %P\n' | awk
'{n=substr($0,12);if(a[$1]==1){print
other,n;}else{a[$1]=1;print first,n;}}'
Yes, I think I have something similar that someone else has used to do
the same thing. Thank you, this is most useful.
One approach in the situation you have, if the filesystem is not corrupt
(which it might be, because files don't create cycles), is to create a
I think I probably hard links to directories. I have observed cpio
going through a loop continously. Since I was doing this on an AIX
JFS filesystem (on an AIX fileserver) it might not have same protections
that I believe Linux when hitting a circular loop.
list of files based on their inode number, and hardlink each file to one
named by its inode number. Just rsync the directory full of inode
numbers. Then re-expand on the destination based on that list.
You should not be following symlinks in a file tree recursion. Rsync,
find, cpio, and others, know not to.
But I suspect some kind of filesystem corruption, or at least some hard
links being applied to directories. The latter can create cycles if not
done carefully (and there is virtually no case to ever do that at all by
intent).
I think this is exactly what's happening. I think I have a number of
cycles that are causing the data to go loopy. (pardon the pun) If
that's the case, how does one find self referential hard/softlinks?
I do not consider it bad organization to have lots of files be
hardlinked. In fact, I have a program that actually seeks out
indentical files and makes them be hardlinked to save space (not
safe in all cases, but safe in most).
Sure, but in a large filesystem, it's been very painful to copy this
data when rsync is taking days instead of hours.
The command find . -type l will only find symlinks. You can find
files that have hard links with find . ! -type d -links +1 -print.
Note that all file types can have hard links, even symlinks. Do
exclude directories as those will have many links for other reasons
(e.g. 1 for self reference, 1 for being inside a directory and 1 each
for each subdirectory within).
Can I also do use find to create a list of files that are not hardlink
and then use --include-file and --exclude=*? I had thought that might
be an alternative way. If I use this rule, does rsync still stat
through the filesystem?
sri
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Skipping hardlinks in a copy
On Thu 08 Mar 2007, Sri Ramkrishna wrote: I think I probably hard links to directories. I have observed cpio going through a loop continously. Since I was doing this on an AIX JFS filesystem (on an AIX fileserver) it might not have same protections that I believe Linux when hitting a circular loop. If there are hard links to directories (apart from the . and .. links) then the filesystem is corrupt; it should be impossible to hardlink a directory to create another entry that points to it. Linux has no protection against such things beyond ensuring the filesystem stays sane... just like AIX most certainly should also have. It's entirely possible that AIX offers some way of hardlinking directories (it's been over 10 years since I last touched AIX :-) but if so, there's no real sane way of handling such situations. I think this is exactly what's happening. I think I have a number of cycles that are causing the data to go loopy. (pardon the pun) If that's the case, how does one find self referential hard/softlinks? It sounds like you're confusing hard links with soft (or symbolic) links, by the way you mention them above. Cpio, find, rsync, whatever will not by default follow symbolic links (unless instructed to do so, in which case any problems arising from that are the user's fault). As I mentioned above, hardlinks to directories shouldn't exist. Without hardlinks to directories, you won't have loops. The command find . -type l will only find symlinks. You can find files that have hard links with find . ! -type d -links +1 -print. Can I also do use find to create a list of files that are not hardlink A file that's not hardlinked will have a link count of 1 (which is very logical if you think about what that link count means...) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Skipping hardlinks in a copy
On Thu, Mar 08, 2007 at 10:15:01PM +0100, Paul Slootman wrote: On Thu 08 Mar 2007, Sri Ramkrishna wrote: I think I probably hard links to directories. I have observed cpio going through a loop continously. Since I was doing this on an AIX JFS filesystem (on an AIX fileserver) it might not have same protections that I believe Linux when hitting a circular loop. If there are hard links to directories (apart from the . and .. links) then the filesystem is corrupt; it should be impossible to hardlink a directory to create another entry that points to it. OK, I wasn't aware that you couldn't hardlink a directory to another directory. Linux has no protection against such things beyond ensuring the filesystem stays sane... just like AIX most certainly should also have. It's entirely possible that AIX offers some way of hardlinking directories (it's been over 10 years since I last touched AIX :-) but if so, there's no real sane way of handling such situations. OK. Looks like I just have to deal wtih each cycle I encounter and break it. Joy. :-) I think this is exactly what's happening. I think I have a number of cycles that are causing the data to go loopy. (pardon the pun) If that's the case, how does one find self referential hard/softlinks? It sounds like you're confusing hard links with soft (or symbolic) links, by the way you mention them above. Cpio, find, rsync, whatever will not by default follow symbolic links (unless instructed to do so, in which case any problems arising from that are the user's fault). As I mentioned above, hardlinks to directories shouldn't exist. Without hardlinks to directories, you won't have loops. I'm at a loss then at what I'm looking at. Maybe it's following symlinks and I have not checked the arguments properly. It might be that symbolic links are causing the issue, but in that case it doesn't seem to explain why it's taking days to copy a level 0 copy. I leaped on the link tree issue, because strace on an rsync was showing it going through the same progression of directories. I'm going to have to go back and run it again and see if I can catch it. The command find . -type l will only find symlinks. You can find files that have hard links with find . ! -type d -links +1 -print. Can I also do use find to create a list of files that are not hardlink A file that's not hardlinked will have a link count of 1 (which is very logical if you think about what that link count means...) Yep, indeed it is. Thank you for taking the time to answer my question, much obliged. sri -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Skipping hardlinks in a copy
On Thu, Mar 08, 2007 at 10:14:39AM -0800, Wayne Davison wrote: On Wed, Mar 07, 2007 at 09:22:08PM -0800, Sriram Ramkrishna wrote: Is there a way to have it skip hard links when doing an rsync? If you mean you want to skip any file that has more than one link, you could do this: find . -type f -links +1 /path/exclude.txt Then, you'd use the exclude.txt file via the --exclude-from option. However, you mentioned loops, and that makes me think that the problem is not with file loops, but dirs looping back in the hierarchy. The above won't help you if that is the case (since find will loop too). Indeed. This is from watching rsync through strace since I wasn't sure why it was taking so long to do an rsync. It seems I need to collect more data to see what's going on. sri -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --link-dest on a local filesystem
On Thu, Mar 08, 2007 at 08:56:54PM +0200, Jaco Kroon wrote: For some strange reason, whenever I recreate the source file, even though it's identical to the old source file --link-dest simply does not create the link. The file isn't identical enough to hard-link unless every preserved detail is identical. So, when using -a (which implies --times, as you noted), the mtime has to be the same for the files to get hard-linked together. However, adding --no-times is not enough because rsync still has to identify the file as unchanged, and that requires the file to have the same size and mtime UNLESS you use the -c (--checksum) option. Using -c, rsync will read each file, comparing the checksum, and then hard-link any matches (as long as you aren't trying to set a conflicting attribute on the destination file, such as mtime, permissions, group, etc.). Helpful hint: Use -i (--itemize-changes) to see why rsync considers a file to be different: if it mentions the file's name in a link-dest copy, the file is not being hard-linked. This sequence of commands should work for you: mkdir a echo foo a/tmp rsync -avi a/ b.1/ touch a/tmp rsync -avic --no-t --link-dest=../b.1 a/ b.0/ ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
question about compare-dest
I have a directory dumps on my laptop containing several dumps of various levels. local-0-2007-03-03.gz local-4-2007-02-12.gz local-4-2007-02-19.gz local-4-2007-02-26.gz local-4-2007-03-05.gz local-5-2007-03-04.gz local-5-2007-03-06.gz local-5-2007-03-07.gz local-5-2007-03-08.gz Naturally the level-0 is the largest and rarely changes. On the target (access.cims.nyu.edu) I have a directory /scratch4/gottlieb/dumps rsync (with --checksum and other options) worked great to push the dumps to access prior to my including a level 0 dump, at which point I exceeded the space available on /scratch4 So I moved the level-0 (on the target only) to /scratch3/gottlieb/dumps and added cmpare-dest=/scratch3/gottlieb to my rsync command as shown below. This may indeed be working correctly, but I noticed that no matter how many -v I use (I tried up to 4) I could not get a confirmation that local-0 was found to agree with the copy on the target, even though I use --checksum. I do see several messages concerning level-0 but not the desired dumps/local-0-2007-03-03.gz is uptodate thanks in advance for any information and a large thanks for rsync. allan rsync --delete --archive --partial --progress --compare-dest=/scratch3/gottlieb --rsync-path=/usr/local/bin/rsync --checksum -v -v -v -v /mnt/hdc7/dumps access.cims.nyu.edu:/scratch4/gottlieb -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [BUG] clear ACL-s on destination
Dnia 8-03-2007 o godz. 15:00 Stanisław Gruszka napisał(a): Destroy ACL-s on destination when no ACL-s differens between source and destination. Wayne, here is example: [EMAIL PROTECTED] /mnt/hda5 $ getfacl --omit-header export/file user::rw- user:apache:-w- group::r-- mask::rw- other::r-- [EMAIL PROTECTED] /mnt/hda5 $ getfacl --omit-header copy/file user::rw- group::r-- other::r-- [EMAIL PROTECTED] /mnt/hda5 $ rsync -av -X -A --delete 127.0.0.1::rsync-export copy receiving incremental file list sent 2859 bytes received 38449 bytes 6355.08 bytes/sec total size is 32768693 speedup is 793.28 [EMAIL PROTECTED] /mnt/hda5 $ getfacl --omit-header copy/file user::rw- user:apache:-w- group::r-- mask::rw- other::r-- [EMAIL PROTECTED] /mnt/hda5 $ rsync -av -X -A --delete 127.0.0.1::rsync-export copy receiving incremental file list sent 308 bytes received 35898 bytes 10344.57 bytes/sec total size is 32768693 speedup is 905.06 [EMAIL PROTECTED] /mnt/hda5 $ getfacl --omit-header copy/file user::rw- group::rw- other::r-- Kelnerka, asystentka, księgowa, informatyk #8211; sprawdź może to właśnie Ciebie poszukują pracodawcy! http://klik.wp.pl/?adr=http%3A%2F%2Fpraca.wp.pl%2Fwyniki_wyszukiwania.htmlsid=1050 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
