Concern: rsync failing to find some attributes in a file transfer?
I have a regular script I run to make static snapshots of my home file system, with each being all the files that changed in the past 24 hours. I just moved my home partition to a new harddisk w/more space. I ran the util and have gotten odd results each time I ran it. This one bothers me... as I'm not sure why the attrs would be missing. How can the names be transfered but no content? Is that possible? Ideas? Thanks! Version info: rsync --version rsync version 3.1.0 protocol version 31 Copyright (C) 1996-2013 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes, prealloc, no SLP uname -a Linux Ishtar 3.15.6-Isht-Van #1 SMP PREEMPT Sat Jul 19 12:31:28 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux File system info: xfs_info /home meta-data=/dev/mapper/Data-Home isize=512agcount=32, agsize=12582896 blks = sectsz=4096 attr=2 data = bsize=4096 blocks=402652672, imaxpct=5 = sunit=16 swidth=16 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=32768, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 --- Command (called from a script file in perl): my $rcmd = [$Rsync]; push( @$rcmd, qw( --8-bit-output --acls --archive --hard-links --human-readable --no-inc-recursive --one-file-system --prune-empty-dirs --whole-file --xattrs ), --compare-dest=$base_lvh-fs_mp/.); output of the program: Rsync with 9 excludes from config file... Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Artists Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Avatars/Production Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/Dragonaut-The Resonance Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/HighSchoolDxD Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/I can't do H Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/Konachan Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/Maria-sama-ga-miteru Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/Miscellaneous Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/SwordArtOnline Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/To Love Ru Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/kiddy grade Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/lastfm Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/reality Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/lib/P/blib Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/lib/mem Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/lib/orig Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/lib/test Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/oldmapdrives Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/reg Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/tmp Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/vbs Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/root/1223/etc/fonts Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/root/1223/etc/local Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/root/1223/etc/samba/save0820/internals Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/root/1223/selinux Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/splunk/bin rsync took 135m, 26s Why would or how would the files and attr-names get transfered but be missing? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options:
Re: Concern: rsync failing to find some attributes in a file transfer?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/26/2014 01:52 AM, L. A. Walsh wrote: I have a regular script I run to make static snapshots of my home file system, with each being all the files that changed in the past 24 hours. I am not clear about the nature of this script. Please provide more details. I just moved my home partition to a new harddisk w/more space. Home Partition? Are we in 1995? Why would you have a partition mounted anywhere other than /boot ? I ran the util and have gotten odd results each time I ran it. What util? What results? This one bothers me... as I'm not sure why the attrs would be missing. Is it really that just extended attributes are missing? You seemed to be in a panic. How can the names be transfered but no content? Is that possible? I am uncertain what this question means. Maybe I have interpreted the rest of your email in the wrong context. Maybe not. I am not sure. Please provide technical details. Ideas? Thanks! Version info: rsync --version rsync version 3.1.0 protocol version 31 Copyright (C) 1996-2013 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes, prealloc, no SLP uname -a Linux Ishtar 3.15.6-Isht-Van #1 SMP PREEMPT Sat Jul 19 12:31:28 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux File system info: xfs_info /home meta-data=/dev/mapper/Data-Home isize=512agcount=32, agsize=12582896 blks = sectsz=4096 attr=2 data = bsize=4096 blocks=402652672, imaxpct=5 = sunit=16 swidth=16 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=32768, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 --- Command (called from a script file in perl): my $rcmd = [$Rsync]; push( @$rcmd, qw( --8-bit-output --acls --archive --hard-links --human-readable --no-inc-recursive --one-file-system --prune-empty-dirs --whole-file --xattrs ), --compare-dest=$base_lvh-fs_mp/.); output of the program: Rsync with 9 excludes from config file... Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Artists Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Avatars/Production Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/Dragonaut-The Resonance Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/HighSchoolDxD Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/I can't do H Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/Konachan Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/Maria-sama-ga-miteru Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/Miscellaneous Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/SwordArtOnline Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/To Love Ru Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/kiddy grade Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/lastfm Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/reality Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/lib/P/blib Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/lib/mem Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/lib/orig Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/lib/test Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/oldmapdrives Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/reg Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/tmp Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/law.V2/bin/vbs Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/root/1223/etc/fonts Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/root/1223/etc/local Missing abbreviated xattr value,
Re: rsync 3.1.0/3.1.1 incompatible with 2.5.7
On Sat, Jul 26, 2014 at 1:49 AM, Wayne Davison way...@samba.org wrote: On Fri, Jul 25, 2014 at 11:50 AM, Ole Tange ta...@gnu.org wrote: : Thanks for the explanation. My question is now: What should I do as a developer? There aren't any particularly easy answers for dealing with an old bit of software that is refusing to play nice with newer software. One possibility is for your wrapping code to do a configure pass that checks the rsync version on both of the hosts and figures out if one is too old (2.6.0 and newer are fine, so only 2.5.7 and older have this issue) and one is too new (3.1.0 is the first to start using protocol 31). If that occurs, you can specify the --protocol=30 option on the newer rsync to avoid the issue. Finding the remote version will be slow. Is there any reason why I don't just look at the local version, and if $version = 3.1.0 then add '--protocol=30'. When will that break down? /Ole -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Concern: rsync failing to find some attributes in a file transfer?
On Sat 26 Jul 2014, Kevin Korb wrote: I just moved my home partition to a new harddisk w/more space. Home Partition? Are we in 1995? Why would you have a partition mounted anywhere other than /boot ? Didn't we just have this discussion already recently? There are valid reasons to have separate filesystems. My /home is encrypted, the rest isn't. I have a separate XFS filesystem as I find that the best option for handling really large files. The relevancy of the separate filesystem to the question isn't clear to me either, so why bring it up? my $rcmd = [$Rsync]; push( @$rcmd, qw( --8-bit-output --acls --archive --hard-links --human-readable --no-inc-recursive --one-file-system --prune-empty-dirs --whole-file --xattrs ), --compare-dest=$base_lvh-fs_mp/.); Transferring with --compare-dest? I thought that the data was being moved from one filesystem to another, that seldomly calls for usage of --compare-dest. It seems to me that the perl script being used is meant for another purpose, and it's being used inappropriately here. Why not just use rsync directly? That way maybe we here on the mailing list can make sense of what's actually happening. Otherwise take it up with the author of that script. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Concern: rsync failing to find some attributes in a file transfer?
Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/26/2014 01:52 AM, L. A. Walsh wrote: I have a regular script I run to make static snapshots of my home file system, with each being all the files that changed in the past 24 hours. I am not clear about the nature of this script. Please provide more details. It's a script that uses the rsync command listed below. It's the rsync command below that that issued the error messages. I just moved my home partition to a new harddisk w/more space. Home Partition? Are we in 1995? Why would you have a partition mounted anywhere other than /boot ? My mom and dad put things on 1 partition, so do many non-computer types. It's not flexible or safe enough for my needs. How would you separate out programs and data? How do you upgrade your OS without destroying your data? How do you implement different backup policies for different types of data? If you want to move your home partition to a different part of the disk or with different make params or even a different file system, how do you do that? When you move your home partition, to a new disk, how do you switch out the home, or media, or whatever partition without rebooting? This isn't MS-DOS or Windows... If you have everything formatted into one partition, how do you make snapshots. If you only have 1 partition, where you do daily backups to? You DO run daily backups, don't you? I ran the util and have gotten odd results each time I ran it. What util? What results? - The results I posted below -- the util.. um... gee, let me think... I'm posting to an rsync list maybe it was visicalc?... nah... rsync! what would I be posting to this list for if this wasn't about rsync? This one bothers me... as I'm not sure why the attrs would be missing. Is it really that just extended attributes are missing? You seemed to be in a panic. Panic would be to my state like like famine to my missing my afternoon snack. Concern!=panic. How can the names be transfered but no content? Is that possible? I am uncertain what this question means. Maybe I have interpreted the rest of your email in the wrong context. Maybe not. I am not sure. Please provide technical details. I thought I did provide the tech details... file system, rsync command that produced it, kernel version. file system params...what more did you have in mind? Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/SwordArtOnline The name trusted.SGI_ACL_DEFAULT is the name of an extended attribute.. For some reason, the name is present in the index of extattrs, but the content associated with that ACL is missing. Another reason for splitting up file systems:... did you notice the execution time at the end: rsync took 135m, 26s. Do you know how long it would take if I added about 20x to that space? What's this about 1995? Do you still have the same data needs now that you did in 95? But all that's apart from the output of the util (that this list is about) with it's version number listed below even! Cripes. Ideas? Thanks! Version info: rsync --version rsync version 3.1.0 protocol version 31 Copyright (C) 1996-2013 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes, prealloc, no SLP uname -a Linux Ishtar 3.15.6-Isht-Van #1 SMP PREEMPT Sat Jul 19 12:31:28 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux File system info: xfs_info /home meta-data=/dev/mapper/Data-Home isize=512agcount=32, agsize=12582896 blks = sectsz=4096 attr=2 data = bsize=4096 blocks=402652672, imaxpct=5 = sunit=16 swidth=16 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=32768, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 --- Command (called from a script file in perl): my $rcmd = [$Rsync]; push( @$rcmd, qw( --8-bit-output --acls --archive --hard-links --human-readable --no-inc-recursive --one-file-system --prune-empty-dirs --whole-file --xattrs ), --compare-dest=$base_lvh-fs_mp/.); output of the program: Rsync with 9 excludes from config file... Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Artists Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Avatars/Production Missing abbreviated xattr value, trusted.SGI_ACL_DEFAULT, for /home.diff/Bliss/Documents/law/Pictures/Scans/Dragonaut-The Resonance
Re: increasing the write block size for high latency
Adam Edgar wrote: It seems the issue is indeed in the ssh layer. scp has the same issue and some work has been done in “fixing” that: http://www.psc.edu/index.php/hpn-ssh From the papers abstract: Status: O SCP and the underlying SSH2 protocol implementation in OpenSSH is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a bottleneck for network throughput of SCP, especially on long and high bandwith network links. It is *A* bottle neck over networks. look for extensions to ssh to ship unencrypted data streams. There's a patch for this @ http://www.psc.edu/index.php/hpn-ssh. However, rsync is dog slow locally as well for exactly the reasons you mention. An extract from another note on this topic (came up on suse list this week). Someone suggested compression for a speed up... I responded to that: On a local copy or local network, that usually slows down transfers. [ 1000:1 speed ratio with large vs. small io sizes):] One might ask why rsync is so slow -- copying 800G from 1 partition to another via xfsdump/restore takes a bit under 2 hours, or about 170MB/s, but with rsync, on the same partition with rsync transfering less than 1/1000th as much (700MB), it took ~70-80 minutes... or about 163kB/s. That's on the same system (local drive - another local drive) Transfer speeds depend on many factors. One of the largest is transfer size (how much transfered with 1 write /read. Transfer 1GB, 1-meg at a time, took 2.08s read, and 1.56s to write (using direct io). Transfer it at 4K: 37.28s, to read, and 43.02s to write. So 20-40x can be accounted for just on R/W size (1k buffers were 4x slower). Many desktop apps still think 4k is a good read size Over a network, causes drop from 500MB/s down to less than 200KB/s (as seen in FF and TB) -- 2500X. Optimal i/o size on my sys is between 16M-256M. So -- to answer your question, MANY things can affect speed, but I'd look at the R/W size first. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Concern: rsync failing to find some attributes in a file transfer?
Kevin Korb wrote: I ran the util and have gotten odd results each time I ran it. What util? What results? Besides the ones I included in the previous email, I ALSO experienced this: (from bug https://bugzilla.samba.org/show_bug.cgi?id=10724): The above was just a toy example designed to illustrate the issue. In practice, rsync 3.1.1 left dozens of such ghost directories inside my --backup-dir. - I ran out of space because of it... creating well over 100,000 empty directories that took up 400M space (on a 600M partition). I thought it might have been a fluke which was why I didn't bother to detail it, but seeing this report -- pretty much cinches it. Copying the command from below as run from my script: my $rcmd = [$Rsync]; push( @$rcmd, qw( --8-bit-output --acls --archive --hard-links --human-readable --no-inc-recursive --one-file-system --prune-empty-dirs --whole-file --xattrs ), --compare-dest=$base_lvh-fs_mp/.); So I am comparing a today snapshot with yesterday's and dumping the difference to a third partition. So that's the other weirdness I was seeing. Do you have a better picture now? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: increasing the write block size for high latency
One thing you that im not seeing factored in is rpm speed of the drives. On 26 Jul 2014 15:05, L. A. Walsh rs...@tlinx.org wrote: Adam Edgar wrote: It seems the issue is indeed in the ssh layer. scp has the same issue and some work has been done in “fixing” that: http://www.psc.edu/index.php/hpn-ssh From the papers abstract: Status: O SCP and the underlying SSH2 protocol implementation in OpenSSH is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a bottleneck for network throughput of SCP, especially on long and high bandwith network links. It is *A* bottle neck over networks. look for extensions to ssh to ship unencrypted data streams. There's a patch for this @ http://www.psc.edu/index.php/hpn-ssh. However, rsync is dog slow locally as well for exactly the reasons you mention. An extract from another note on this topic (came up on suse list this week). Someone suggested compression for a speed up... I responded to that: On a local copy or local network, that usually slows down transfers. [ 1000:1 speed ratio with large vs. small io sizes):] One might ask why rsync is so slow -- copying 800G from 1 partition to another via xfsdump/restore takes a bit under 2 hours, or about 170MB/s, but with rsync, on the same partition with rsync transfering less than 1/1000th as much (700MB), it took ~70-80 minutes... or about 163kB/s. That's on the same system (local drive - another local drive) Transfer speeds depend on many factors. One of the largest is transfer size (how much transfered with 1 write /read. Transfer 1GB, 1-meg at a time, took 2.08s read, and 1.56s to write (using direct io). Transfer it at 4K: 37.28s, to read, and 43.02s to write. So 20-40x can be accounted for just on R/W size (1k buffers were 4x slower). Many desktop apps still think 4k is a good read size Over a network, causes drop from 500MB/s down to less than 200KB/s (as seen in FF and TB) -- 2500X. Optimal i/o size on my sys is between 16M-256M. So -- to answer your question, MANY things can affect speed, but I'd look at the R/W size first. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/ mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: increasing the write block size for high latency
Jonathan Aquilina wrote: One thing you that im not seeing factored in is rpm speed of the drives. Since my tests are run on the same machines and drives, such things factor out (as do cpu Hz, memory speeds, controller firmware, ... etc). Make sense? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 3.1.0/3.1.1 incompatible with 2.5.7
On Fri, Jul 25, 2014 at 11:43 PM, Ole Tange ta...@gnu.org wrote: Finding the remote version will be slow. Is there any reason why I don't just look at the local version, and if $version = 3.1.0 then add '--protocol=30'. When will that break down? It won't break -- you'll just miss out on the protocol 31 improvements that would occur between 3.1.x clients. For instance, nsec timestamp preservation, deleted/created info in the --stats, improved exit message propagation, etc. You may well find that trade-off acceptable, at least for now. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Concern: rsync failing to find some attributes in a file transfer?
On Fri, Jul 25, 2014 at 10:52 PM, L. A. Walsh rs...@tlinx.org wrote: Why would or how would the files and attr-names get transfered but be missing? Give 3.1.1 a try -- it has a fix in it for miss-sorted attr names when running as non-root. Alternately, try running (at least the receiving side) as root. Here's the NEWS entry for this fix: - Fixed a bug in the xattr-finding code that could make a non-root-run receiver not able to find some xattr numbers. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: increasing the write block size for high latency
On Tue, Jul 8, 2014 at 8:07 AM, Adam Edgar aed...@research.att.com wrote: It seems the issue is indeed in the ssh layer. scp has the same issue and some work has been done in “fixing” that: http://www.psc.edu/index.php/hpn-ssh From the papers abstract: SCP and the underlying SSH2 protocol implementation in OpenSSH is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a bottleneck for network throughput of SCP, especially on long and high bandwith network links. It's a bit hard to believe that PSC's performance patches still haven't been merged. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Concern: rsync failing to find some attributes in a file transfer?
Wayne Davison wrote: On Fri, Jul 25, 2014 at 10:52 PM, L. A. Walsh rs...@tlinx.org mailto:rs...@tlinx.org wrote: Why would or how would the files and attr-names get transfered but be missing? Give 3.1.1 a try -- it has a fix in it for miss-sorted attr names when running as non-root. Alternately, try running (at least the receiving side) as root. Here's the NEWS entry for this fix: - Fixed a bug in the xattr-finding code that could make a non-root-run receiver not able to find some xattr numbers. Since it was generating a volume snapshot, it was already running as root -- and it was a local - local copy. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: increasing the write block size for high latency
On Jul 26, 2014, at 2:53 PM, Dan Stromberg drsali...@gmail.com wrote: On Tue, Jul 8, 2014 at 8:07 AM, Adam Edgar aed...@research.att.com wrote: It seems the issue is indeed in the ssh layer. scp has the same issue and some work has been done in “fixing” that: http://www.psc.edu/index.php/hpn-ssh From the papers abstract: SCP and the underlying SSH2 protocol implementation in OpenSSH is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a bottleneck for network throughput of SCP, especially on long and high bandwith network links. It's a bit hard to believe that PSC's performance patches still haven't been merged. Not really. The concern has to be about using malloc. You can open lots of security holes if you are not careful with dynamic allocation. Larger static buffers would be easier to check for exploits but would limit the scalability. ASE -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Concern: rsync failing to find some attributes in a file transfer?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I wasn't objecting to the use of multiple file systems. I have a bunch of them too. I was objecting to the use of partitions to achieve multiple files systems. Logical volume management has been available for a long time and now we also have access to file systems that include such features. On 07/26/2014 04:06 AM, Paul Slootman wrote: On Sat 26 Jul 2014, Kevin Korb wrote: I just moved my home partition to a new harddisk w/more space. Home Partition? Are we in 1995? Why would you have a partition mounted anywhere other than /boot ? Didn't we just have this discussion already recently? There are valid reasons to have separate filesystems. My /home is encrypted, the rest isn't. I have a separate XFS filesystem as I find that the best option for handling really large files. The relevancy of the separate filesystem to the question isn't clear to me either, so why bring it up? my $rcmd = [$Rsync]; push( @$rcmd, qw( --8-bit-output --acls --archive --hard-links --human-readable --no-inc-recursive --one-file-system --prune-empty-dirs --whole-file --xattrs ), --compare-dest=$base_lvh-fs_mp/.); Transferring with --compare-dest? I thought that the data was being moved from one filesystem to another, that seldomly calls for usage of --compare-dest. It seems to me that the perl script being used is meant for another purpose, and it's being used inappropriately here. Why not just use rsync directly? That way maybe we here on the mailing list can make sense of what's actually happening. Otherwise take it up with the author of that script. Paul - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlPUEM8ACgkQVKC1jlbQAQcuWgCfW6bqFMXNbC9dX8ZadtqZB0cF IEYAn2zzfWlOySKPrzn4DjSc7ElUc4he =9LaA -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Concern: rsync failing to find some attributes in a file transfer?
On 07/26/2014 03:34:23 PM, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I wasn't objecting to the use of multiple file systems. I have a bunch of them too. I was objecting to the use of partitions to achieve multiple files systems. Logical volume management has been available for a long time and now we also have access to file systems that include such features. I too like logical volume management but that does not mean it's right for everyone. E.g. chasing badspot block numbers back and forth between the underlying media and the file system makes me cranky. Karl k...@meme.com Free Software: You don't pay back, you pay forward. -- Robert A. Heinlein -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Concern: rsync failing to find some attributes in a file transfer?
On Fri, Jul 25, 2014 at 11:35 PM, Kevin Korb k...@sanitarium.net wrote: I just moved my home partition to a new harddisk w/more space. Home Partition? Are we in 1995? Why would you have a partition mounted anywhere other than /boot ? That's a bit harsh, particularly considering that having a /home partition never really stopped being useful. Having a separate partition for user files means that if one (of potentially many) user(s) fills up the /home partition, the machine's OS files don't get messed up when written to, as a painful side effect. Both kinds of write failures are bad, but they needn't be concomitant. Also, some people might just have multiple disks in the same system; or be using a network filesystem like NFS, CIFS, sshfs, c. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
