Rsyncing huge file, timeout on rsyncd
Hello, let me describe my setup. Source server is debian wheezy, rsync 3.0.9 Destination is qnap TS-410, with rsyncd enabled, rsync 3.0.7 I'm trying to rsync two files, which are exported from source server as iscsi targets (windows iscsi backups). Files are on btrfs, so I use snapshots during rsync, so files don't change. One file is 50GB, second is 550GB in size. Initial sync over rsyncd daemon went fine. Now, i'm trying to sync data, after change. When i try to sync over rsyncd, it ends after 20 minutes with: rsync: connection unexpectedly closed (2438836 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] When i try to rsync over ssh, it takes ages to show progress status (around 1 hour for smaller file, 4-5 hours for big file). Whats going out there? Source server is idle during this no progress period, destination Qnap has one rsync process eating cpu. My commands are (example with one file) for rsyncd rsync --inplace --progress --numeric-ids --times --password-file .passwd iscsi/iscsiDisk1 rsync://backup-rsync@192.168.5.250:/iSCSI-backup/iscsiDisk1 for ssh rsync --inplace --progress --numeric-ids --times iscsi/iscsiDisk1 admin@192.168.5.250:/share/MD0_DATA/iSCSI-backup/iscsiDisk1 With regards Libor Klepac signature.asc Description: This is a digitally signed message part. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsyncing huge file, timeout on rsyncd
Just to clarify When using rsyncd script, there is also no progress from beginning, it just timeouts Libor Dne Thursday 02 August 2012 12:26:13, Libor Klepáč napsal(a): Hello, let me describe my setup. Source server is debian wheezy, rsync 3.0.9 Destination is qnap TS-410, with rsyncd enabled, rsync 3.0.7 I'm trying to rsync two files, which are exported from source server as iscsi targets (windows iscsi backups). Files are on btrfs, so I use snapshots during rsync, so files don't change. One file is 50GB, second is 550GB in size. Initial sync over rsyncd daemon went fine. Now, i'm trying to sync data, after change. When i try to sync over rsyncd, it ends after 20 minutes with: rsync: connection unexpectedly closed (2438836 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] When i try to rsync over ssh, it takes ages to show progress status (around 1 hour for smaller file, 4-5 hours for big file). Whats going out there? Source server is idle during this no progress period, destination Qnap has one rsync process eating cpu. My commands are (example with one file) for rsyncd rsync --inplace --progress --numeric-ids --times --password-file .passwd iscsi/iscsiDisk1 rsync://backup-rsync@192.168.5.250:/iSCSI-backup/iscsiDisk1 for ssh rsync --inplace --progress --numeric-ids --times iscsi/iscsiDisk1 admin@192.168.5.250:/share/MD0_DATA/iSCSI-backup/iscsiDisk1 With regards Libor Klepac signature.asc Description: This is a digitally signed message part. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsyncing huge file, timeout on rsyncd
--timeout=SECONDS set I/O timeout in seconds I think this timeout must be set big enough so that data actually gets transferred during the window. Having the target verify that nothing has changed yet seems to not qualify as resetting the timeout. Figure on enough time so that the target can read the entire target file. (Most likely there is a wee bit of change at the very end of the files) From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Libor Klepác Sent: Thursday, August 02, 2012 5:47 AM To: rsync@lists.samba.org Subject: Re: Rsyncing huge file, timeout on rsyncd Just to clarify When using rsyncd script, there is also no progress from beginning, it just timeouts Libor Dne Thursday 02 August 2012 12:26:13, Libor Klepáč napsal(a): Hello, let me describe my setup. Source server is debian wheezy, rsync 3.0.9 Destination is qnap TS-410, with rsyncd enabled, rsync 3.0.7 I'm trying to rsync two files, which are exported from source server as iscsi targets (windows iscsi backups). Files are on btrfs, so I use snapshots during rsync, so files don't change. One file is 50GB, second is 550GB in size. Initial sync over rsyncd daemon went fine. Now, i'm trying to sync data, after change. When i try to sync over rsyncd, it ends after 20 minutes with: rsync: connection unexpectedly closed (2438836 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] When i try to rsync over ssh, it takes ages to show progress status (around 1 hour for smaller file, 4-5 hours for big file). Whats going out there? Source server is idle during this no progress period, destination Qnap has one rsync process eating cpu. My commands are (example with one file) for rsyncd rsync --inplace --progress --numeric-ids --times --password-file .passwd iscsi/iscsiDisk1 rsync://backup-rsync@192.168.5.250:/iSCSI-backup/iscsiDisk1 for ssh rsync --inplace --progress --numeric-ids --times iscsi/iscsiDisk1 admin@192.168.5.250:/share/MD0_DATA/iSCSI-backup/iscsiDisk1mailto:admin@192.168.5.250:/share/MD0_DATA/iSCSI-backup/iscsiDisk1 With regards Libor Klepac -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsyncing huge file, timeout on rsyncd
Ok, i'l try setting timeout, but according to documentation, default is timeout=0 which means no timeout. Maybe setting --contimeout to some large value should help? Libor Dne Thursday 02 August 2012 07:03:51, Tony Abernethy napsal(a): --timeout=SECONDS set I/O timeout in seconds I think this timeout must be set big enough so that data actually gets transferred during the window. Having the target verify that nothing has changed yet seems to not qualify as resetting the timeout. Figure on enough time so that the target can read the entire target file. (Most likely there is a wee bit of change at the very end of the files) From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Libor Klepác Sent: Thursday, August 02, 2012 5:47 AM To: rsync@lists.samba.org Subject: Re: Rsyncing huge file, timeout on rsyncd Just to clarify When using rsyncd script, there is also no progress from beginning, it just timeouts Libor Dne Thursday 02 August 2012 12:26:13, Libor Klepáč napsal(a): Hello, let me describe my setup. Source server is debian wheezy, rsync 3.0.9 Destination is qnap TS-410, with rsyncd enabled, rsync 3.0.7 I'm trying to rsync two files, which are exported from source server as iscsi targets (windows iscsi backups). Files are on btrfs, so I use snapshots during rsync, so files don't change. One file is 50GB, second is 550GB in size. Initial sync over rsyncd daemon went fine. Now, i'm trying to sync data, after change. When i try to sync over rsyncd, it ends after 20 minutes with: rsync: connection unexpectedly closed (2438836 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] When i try to rsync over ssh, it takes ages to show progress status (around 1 hour for smaller file, 4-5 hours for big file). Whats going out there? Source server is idle during this no progress period, destination Qnap has one rsync process eating cpu. My commands are (example with one file) for rsyncd rsync --inplace --progress --numeric-ids --times --password-file .passwd iscsi/iscsiDisk1 rsync://backup-rsync@192.168.5.250:/iSCSI-backup/iscsiDisk1 for ssh rsync --inplace --progress --numeric-ids --times iscsi/iscsiDisk1 admin@192.168.5.250:/share/MD0_DATA/iSCSI-backup/iscsiDisk1 With regards Libor Klepac signature.asc Description: This is a digitally signed message part. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsyncing huge file, timeout on rsyncd
On Thu 02 Aug 2012, Libor Klepáč wrote: Ok, i'l try setting timeout, but according to documentation, default is timeout=0 which means no timeout. IIRC setting the timeout explicitly on both ends (if you're using an rsync daemon) to the same value will cause some sort of heartbeat activity. Note also that an inferior firewall may also forget the connection if there is no traffic for a longer period of time. Maybe setting --contimeout to some large value should help? No, I'm quite sure that has nothing to do with this; you _had_ a connection, which timed out. --contimeout is for the timeout for making the initial connection. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsyncing huge file, timeout on rsyncd
Hello, tried to set timeout=7200 on both sides also enabled logging in rsyncd Here is output from server 2012/08/02 15:07:23 [2339] connect from ms-backup.xx.local (192.168.5.213) 2012/08/02 15:07:23 [2339] rsync to iSCSI-backup/ from backup-rsync@ms- backup.xx.local (192.168.5.213) 2012/08/02 13:07:23 [2339] receiving file list 2012/08/02 13:27:26 [2339] rsync error: timeout in data send/receive (code 30) at io.c(137) [receiver=3.0.7] 2012/08/02 13:27:27 [2339] rsync: connection unexpectedly closed (95 bytes received so far) [generator] 2012/08/02 13:27:27 [2339] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [generator=3.0.7] Here is output from client opening tcp connection to 192.168.5.250 port 873 sending daemon args: --server -vvte.Lsf --timeout=7200 --numeric-ids --inplace . iSCSI-backup/ rsync: connection unexpectedly closed (2933968 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] real20m3.942s user0m0.064s sys 0m0.156s Command line was time rsync -vv --contimeout=7200 --timeout=7200 --inplace --progress -- numeric-ids --times --password-file .passwd iscsi/iscsiDisk1 rsync://backup- rsync@192.168.5.250:/iSCSI-backup/ Libor Dne Thursday 02 August 2012 14:20:19, Paul Slootman napsal(a): On Thu 02 Aug 2012, Libor Klepáč wrote: Ok, i'l try setting timeout, but according to documentation, default is timeout=0 which means no timeout. IIRC setting the timeout explicitly on both ends (if you're using an rsync daemon) to the same value will cause some sort of heartbeat activity. Note also that an inferior firewall may also forget the connection if there is no traffic for a longer period of time. Maybe setting --contimeout to some large value should help? No, I'm quite sure that has nothing to do with this; you _had_ a connection, which timed out. --contimeout is for the timeout for making the initial connection. Paul signature.asc Description: This is a digitally signed message part. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Cache file list in daemon mode?
Hello. I suspect that what I want to do is not possible with rsync, but this is the best place to double-check. We are pushing files to a remote target that stores them on a very slow network file system. There are also over a million files on the target. Consequently, running rsync to push an update takes hours while the remote side enumerates and stats all those files. I thought, that wouldn't be necessary if the remote side was running rsync in daemon mode, and that it only built its internal map of the files there once after startup, thereafter updating that in-memory list with every push it receives. For that to work, there would have to be some flag I could set to promise rsyncd that no files in the target would be updated through any means other than rsyncd. I looked for such an option, did some experimenting with write-only targets, straced the daemon and saw I wasn't getting anywhere. Our alternatives are options like building a list of what has changed and copying only those things across. There's some housekeeping and race condition avoidance in there that makes it more than a couple of simple commands and replicates some of what rsync knows how to do already. By any chance is what I want to do possible with rsyncd, or some other tool? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Cache file list in daemon mode?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 is it possible to talk directly to the NFS server via rsyncd or rsync over ssh? Eliminating the extra hop through a network mount should make a big difference. On 08/02/12 18:30, Peter Scott wrote: Hello. I suspect that what I want to do is not possible with rsync, but this is the best place to double-check. We are pushing files to a remote target that stores them on a very slow network file system. There are also over a million files on the target. Consequently, running rsync to push an update takes hours while the remote side enumerates and stats all those files. I thought, that wouldn't be necessary if the remote side was running rsync in daemon mode, and that it only built its internal map of the files there once after startup, thereafter updating that in-memory list with every push it receives. For that to work, there would have to be some flag I could set to promise rsyncd that no files in the target would be updated through any means other than rsyncd. I looked for such an option, did some experimenting with write-only targets, straced the daemon and saw I wasn't getting anywhere. Our alternatives are options like building a list of what has changed and copying only those things across. There's some housekeeping and race condition avoidance in there that makes it more than a couple of simple commands and replicates some of what rsync knows how to do already. By any chance is what I want to do possible with rsyncd, or some other tool? - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlAbApcACgkQVKC1jlbQAQcYHgCg6HYJATQn4kf/duAC+owxPq5w TNAAoNFXN8MAcKbBEbXk2cKgF/4MNgE5 =D9DM -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Cache file list in daemon mode?
We're considering that, but it's Gluster, not NFS, and it's peer-to-peer, not client-server. Options in that direction start getting more complicated than the 'find -mtime... scp' approach pretty fast. On 8/2/2012 3:43 PM, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 is it possible to talk directly to the NFS server via rsyncd or rsync over ssh? Eliminating the extra hop through a network mount should make a big difference. On 08/02/12 18:30, Peter Scott wrote: Hello. I suspect that what I want to do is not possible with rsync, but this is the best place to double-check. We are pushing files to a remote target that stores them on a very slow network file system. There are also over a million files on the target. Consequently, running rsync to push an update takes hours while the remote side enumerates and stats all those files. I thought, that wouldn't be necessary if the remote side was running rsync in daemon mode, and that it only built its internal map of the files there once after startup, thereafter updating that in-memory list with every push it receives. For that to work, there would have to be some flag I could set to promise rsyncd that no files in the target would be updated through any means other than rsyncd. I looked for such an option, did some experimenting with write-only targets, straced the daemon and saw I wasn't getting anywhere. Our alternatives are options like building a list of what has changed and copying only those things across. There's some housekeeping and race condition avoidance in there that makes it more than a couple of simple commands and replicates some of what rsync knows how to do already. By any chance is what I want to do possible with rsyncd, or some other tool? - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlAbApcACgkQVKC1jlbQAQcYHgCg6HYJATQn4kf/duAC+owxPq5w TNAAoNFXN8MAcKbBEbXk2cKgF/4MNgE5 =D9DM -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Cache file list in daemon mode?
So what you're really saying is gluster is quite slow at doing recursive directory listings, so how about just using find on the real backend bricks to find the files that have changed since last run, merge those listings together (to get rid of dupes) and then get rsync to just update those against the gluster service? -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
