multiplexing overflow errors
I'm using rsync 3.0.6 between two solaris 8/10 hosts an I frequently get multiplexing overflow errors: log/20091123_223901/dresden.log:multiplexing overflow 101:7104843 [sender] log/20091123_223901/frankfurt.log:multiplexing overflow 101:7104843 [sender] log/20091123_223901/hannover.log:multiplexing overflow 101:7104843 [sender] log/20091123_223901/muenchen.log:multiplexing overflow 101:7104843 [sender] log/20091123_223901/stuttgart.log:multiplexing overflow 101:7104843 [sender] log/20091124_193051/dresden.log:multiplexing overflow 101:7104843 [sender] log/20091124_193051/muenchen.log:multiplexing overflow 101:7104843 [sender] I have to admit that I have to syncronize HUGE directories with up to 1 000 000 tiny files! But this does not fail in general. After a few retries it works, but it is annoying. Any clue? I compiled the source myself, so I may be able to debug it or to include some diagnostic messages to narrow the problem. My current version is: rsync version 3.0.6 protocol version 30 Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, no IPv6, batchfiles, inplace, append, ACLs, no xattrs, iconv, no symtimes rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. Dieter -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--copy-links and --hard-links
Hi, I want to use rsync in a may be unusual way: I have a source tree containing lots of symbolic links and I use the option --copy-links to get the physical files (the referents of the symlinks) on the target host. As the host uses the synchronized files in a read-only fashion, I also want to get hardlinks for all identical files, to save space. Thus I also use --hard-links. Because managing hardlinks is hard to handle (some lookup table is needed), only inodes with a hardlink count 1 are considered (flink.c:1250 st.st_nlink 1). Unfortunately most of my symlinked files have a link count of 1 and thus they are copied twice. If, however, the source file accidentally has a link count 1 (which is not related to the multiple symlinks) rsync will create a hard link on the target host, as supposed. To handle this situation I simply disabled the (st.st_nlink 1) filter if copy_links is true, too. This seems to work for me, but I don't know if this is a sufficient solution for others. May be a special option should be considered to disable the (st.st_nlink 1) optimization explicitly. Or my situation is too exotic to be considered to fix. Dieter. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
synchronizing hard linked trees
Hi, I have to synchronize a directory tree of one machine with a tree on a remote machine. The tree on the remote machine is used read only. In addition I clone the current target tree before each synchronizing using hardlinks to be able to switch back to any previous version if the synchronization failed or if it contained any bad data. This works since 2002 using tar and a perl script. After I got a direct network connection with ssh, I want to switch to rsync instead. At first sight rsync seems to be ideal, but there remain a few questions to me: a) avoiding inplace modifications: As each target file is possibly hardlinked to some backup version, i never want to perform any incremental backup modifying existing files. Is this guaranteed if I don't use --inplace? Or is there something like --no-inplace? b) mixing symlinks and hardlinks: There may be additional hardlinks within the transfer set. This is handled properly by --hard-links, but in my case it is a little bit more complicated: The source tree also contains symlinks which are to be dissolved using --copy-links. If three symlinks are pointing to the same source file, the file should be copied once and hardlinked on the target system. Unfortunately rsync does not recognize this situation as long as the (hard) link count of the source inode is 1. In this case the file is not considered as a candidate for target hardlink, and is transferred three times instead. (In my case this does not happen frequently as almost all of my files are excessively hardlinked anyway.) c) using time stamps: An other problem arises with timestamps: Most of my files have synchronized timestamps, but some few may differ. All files with equal size and timestamp are considered equal, so I want to use --times. If, however, the time stamp differs, the file is synchronized (transferred) without any further check. Would it be possible to do an additional content check to verify, if the files content really differ, and to leave the target file untouched, if they differ only by their time stamps, not by content? This is not only a matter of performance, but of the integrity of my target tree. Thanks for any comment, Dieter.-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
