link(2) EMLINK error behavior with --link-dest and --hard-links

Hi, I'm having a problem using --link-dest and --hard-links when the fs hits the hard link limit (link(2) returns EMLINK). Using rsync 3.0.7 an error is thrown and the target file is not created. Glancing at git head it _looks_ like things could now be a little nicer. Perhaps the target file

Re: link(2) EMLINK error behavior with --link-dest and --hard-links

Should I submit a bug? Will you take a patch? On 03/24/2012 11:11:47 PM, Karl O. Pinc wrote: Hi, I'm having a problem using --link-dest and --hard-links when the fs hits the hard link limit (link(2) returns EMLINK). Using rsync 3.0.7 an error is thrown and the target file is not created

Re: Run rsync even not connected

On 04/11/2012 09:23:57 PM, Kevin Korb wrote: Cron will allow for an rsync to be running in the background. However, there are additional steps (such as a lock file) you should take to make sure two don't end up running at the same time. You could also use at if you want something to run once,

Re: Rsync takes long time to finish

On 04/12/2012 04:36:44 PM, vijay patel wrote: We are running Kernel 2.6.18-308.1.1.el5 which is latest in RHEL 5.8 on both the server. I think i might have to explore option of using ext4. Before you do anything you want to figure out why it is slow so you can solve the real problem.

Re: Getting messages failed to stat only in cron execution

rsync: recv_generator: failed to stat /mnt/tmp/backup/museum/documenten/secretariaat/vrijwilligers/ medewerkersboek en poster/losse foto's medewerkers/oude foto's medewerkers die we willen bewaren/wienk, ari\#303\#253l .jpg: Invalid or incomplete multibyte or wide character (84)

Re: Using rsync to mirror directories where root owns file, using non-root user to initiate session

On 06/20/2012 05:29:09 PM, Kevin Korb wrote: http://www.sanitarium.net/rsyncfaq/#sudo Along these lines... Somehow or another you need root access on the remote side in order to properly set permissions. You can use ssh public keys to invoke a rsync daemon. In /root/.ssh/authorized_keys you put

Re: Using rsync to mirror directories where root owns file, using non-root user to initiate session

On 06/20/2012 10:40:57 PM, Kevin Korb wrote: On 06/20/12 21:53, Karl O. Pinc wrote: On 06/20/2012 05:29:09 PM, Kevin Korb wrote: Somehow or another you need root access on the remote side in order to properly set permissions. Not permissions, ownership. Quite right. I shouldn't

Re: Hlink node data for 2282618 already has path=...

On 06/24/2012 10:40:11 AM, Brian J. Murrell wrote: So, like a lot of people, I am using --link-dest to do backups. My backup target is ext4 so with a hard link limit of 64K. I do end up with trying to create too many links at some point though and get the following sequence of events: snip

Re: BDsync: Block Device sync

On 06/27/2012 06:51:29 AM, Rolf Fokkens wrote: After having wrestled with rsync and several patches I found a solution to synchronize block devices: BDsync. Bdsync can be used to synchronize block devices over a network. It generates a binary diff in an efficient way by comparing MD5

Re: BDsync: Block Device sync

On 06/27/2012 08:48:30 AM, Karl O. Pinc wrote: On 06/27/2012 06:51:29 AM, Rolf Fokkens wrote: After having wrestled with rsync and several patches I found a solution to synchronize block devices: BDsync. Bdsync can be used to synchronize block devices over a network. It generates

Is --sparse suitable for general purpose use?

Hi, I'm using rsync with --link-dest to do backups. I don't have any sparse files, but someday I might. Should I be using --sparse? I notice that -S is not implied by -a. This makes me suspicious that --sparse is not (yet?) suitable for general purpose use. There also seem to be outstanding

Re: rsync hashing / collision handling?

On 10/02/2012 03:58:04 PM, Andrew Pennebaker wrote: Have recent versions of rsync considered using a more robust implementation of file change monitoring, like Dropbox https://www.dropbox.com/help/8/en does? The above link says Dropbox does a binary diff. How are you proposing to do a binary

Re: Problem rsyncing 450GB file to my NAS: 'connection unexpectedly closed'

On 10/12/2012 01:06:03 PM, Justin T Pryzby wrote: It sounds like a daemon may be timing out; is there a timeout specified in rsyncd.conf? Is there a remote logfile with any useful content? I've gotten a 12 error code when a lame firewall broke the connection. Karl k...@meme.com Free

Re: rsync has big problem with leaving lots of empty dirs around...

On 10/19/2012 03:39:18 PM, Linda Walsh wrote: Karl O. Pinc wrote: If you want help with your problem you're going to have to describe exactly what you're doing; what command you're running, what version of rsync, and so forth. --- Thanks...was going to file a bug report on it when I got

Re: vanilla rsync 3.0.9 hangs after transferring ~2000 files

On 11/02/2012 08:33:13 AM, Christian Iversen wrote: Hello rsync folks Do you have any ideas what the problem might be? Or how I can help debug it? I have no idea regards the problem, but it never hurts to do a tcpdump (-w, possibly -s) and look at what's on the wire. Perhaps you can

Re: vanilla rsync 3.0.9 hangs after transferring ~2000 files

On 11/02/2012 08:33:13 AM, Christian Iversen wrote: Hello rsync folks However, 1 server is giving me a lot of trouble. It has a directory with (currently) 734088 files in it, and every time I try to backup this dir, rsync hangs after transferring roughly 2000 files. Since it might be

Re: vanilla rsync 3.0.9 hangs after transferring ~2000 files

On 11/08/2012 05:42:08 PM, Christian Iversen wrote: It's vanilla ext4, with: $ df -i /data FilesystemInodes IUsed IFree IUse% Mounted on /dev/mapper/storage-data 16384000 1475262 14908738 10% /data and: $ file /dev/mapper/storage-data

Re: rsync in conjunction with the --link-dest option does not output deleted files

On 01/08/2013 07:17:15 AM, Kevin Korb wrote: You might find this useful: http://sanitarium.net/unix_stuff/rspaghetti_backup/diff_backup.pl.txt It is a script I wrote that diffs 2 backup directories and will tell you what is missing between them. There's always some simple shell (bash)

Re: Does rsync need a --ignore-unreadable-files option?

Hi, On 01/10/2013 08:07:18 PM, Allen Supynuk wrote: It strikes me though, that what I really want is an rsync option '--ignore-files-with-no-read-perms'. While you're at it, why not a more generic --ignore-errno foo where foo is (a defined subset of) the Posix error numbers listed in

Re: Does rsync need a --ignore-unreadable-files option?

On 01/11/2013 09:08:52 AM, Allen Supynuk wrote: While you're at it, why not a more generic --ignore-errno foo where foo is (a defined subset of) the Posix error numbers listed in errno(3). The names, alternately, could be used. So your application would use --ignore-errno EACCES

Re: Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

On 01/23/2013 02:15:06 AM, Voelker, Bernhard wrote: Kevin Korb wrote: On 01/22/13 18:12, Kevin Korb wrote: That is the old way that pre-dates --link-dest. Instead of cp - al daily.02 daily.01 you can do a mkdir daily.01 then an rsync ... --link-dest=../daily.02 daily.01 I'm

Re: Re: rsyncssl

On 01/28/2013 04:24:13 AM, devz...@web.de wrote: The only place that an SSL would make some sense, is if you are going to do it to/from an rsync daemon, yes, exactly. but then how would that be better than a ssh-only account with keys/etc. only allowing the rsync to execute?

Re: Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

Here's my rsync based backup system. http://wikisend.com/download/377440/rsync_backup-0.26.tar.gz It's an rsync based backup system utilizing hard links to reduce storage requirements. It supports both push and pull. It uses public keys with ssh for the transport. It works and I've used it

A --exclude-checksum option?

Hi, I use rsync with hardlinks for backup, once a week doing checksums to ensure there's no filesystem corruption in the backed-up data. I also use tmpwatch, or something similar, to clean up /tmp, it removes files that have not been accessed recently. (atime older than some configured limit).

Re: A --exclude-checksum option?

. (Presumably the backup side hardware/ fs will be fixed quickly.) And I don't care that --checksum means that the rsync takes longer once a week. Sounds like you're leaning toward it's a niche feature and let's not clutter up rsync (further). On 02/12/13 15:42, Karl O. Pinc wrote: Hi, I use

Re: A --exclude-checksum option?

On 02/17/2013 01:59:02 PM, Wayne Davison wrote: On Tue, Feb 12, 2013 at 12:42 PM, Karl O. Pinc k...@meme.com wrote: It occurs to me that a handy solution might be to have an rsync option, similar to the --exclude option, which would allow checksumming to happen throughout most

Re: High Speed WAN Rsync now possible!!!

On 02/22/2013 09:54:03 AM, Erich Weiler wrote: Folks- Just wanted to plug a totally awesome software package from a group I know: UDR (UDT Enabled Rsync). For those not familiar with UDT, it is a low level network protocol based on UDP that allows for high speed transfers over high

Re: contribute to rsync

On 05/19/2013 10:19:10 PM, garvit sharma wrote: Hello All, Myself Garvit Sharma working in hyderabad central university(HCU) at DCIS. I have started using rsync to sync the data and i found it very interesting. After using rsync a lot i realized to contribute to rsync by

Re: Need hint for my question regarding the working of rsync.

On 11/12/2013 03:50:20 PM, Wayne Davison wrote Yes, the receiver sends all the checksums that it generates at once For really big files it would be interesting to amend this rule to one where the sending side waits only long enough for a certain number of checksums to arrive before it

Re: Need hint for my question regarding the working of rsync.

On 11/12/2013 04:13:01 PM, Karl O. Pinc wrote: On 11/12/2013 03:50:20 PM, Wayne Davison wrote Yes, the receiver sends all the checksums that it generates at once For really big files it would be interesting to amend this rule to one where the sending side waits only long enough

Re: Need hint for my question regarding the working of rsync.

On 11/13/2013 12:03:21 PM, Kevin Korb wrote: OK, in the case of using v3 with --link-dest and not --checksum most of the initial activity on the sender would be doing calls to stat() to index what is there. The receiving side would be doing 2x the stat() calls (you have 2 --link-dest dirs

Re: Need hint for my question regarding the working of rsync.

necessarily 11/13/2013 01:04:29 PM, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Is there a hard links limit? I have been in the 70-80 million range on ext4 without a problem (other than performance which is why I switched to ZFS for that use case). It's a per-file

Re: silent data corruption with rsync

On 03/11/2014 11:02:28 AM, Sig Pam wrote: Hi everbody! I'm currently working in a project which has to copy huge amounts of data from one storage to another. For a reason I cannot validate any longer, there is a roumor that rsync may silently corrupt data. Personally, I don't believe that.

Re: Alternative rsync client for Windows

So long as your compiling a list of things to overcome to restore a MS Windows box you need to add the following: You'd have to have an (exact?) copy of the original hardware onto which you wish to restore. I try to stay away from Windows so am no expert, but I believe that the OEMs are the ones

Re: [Bug 10637] rsync --link-dest should break hard links when encountering Too many links

On 06/06/2014 12:08:46 AM, L. A. Walsh wrote: samba-b...@samba.org wrote: https://bugzilla.samba.org/show_bug.cgi?id=10637 --- Comment #1 from Karl O. Pinc k...@meme.com 2014-05-28 19:05:04 UTC --- Yum is also rsync happy. That's where our --link-dest backups always break due

Re: Concern: rsync failing to find some attributes in a file transfer?

On 07/26/2014 03:34:23 PM, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I wasn't objecting to the use of multiple file systems. I have a bunch of them too. I was objecting to the use of partitions to achieve multiple files systems. Logical volume management has been

Re: Using --fuzzy

On 11/16/2014 03:53:12 PM, Joe wrote: I have a lot of files (and directories) (up to a few hundred at a time) that I get from various sources. Some time after I get them (after they are already backed up), I often have to move them around and normalize their names. When I do this, rsync

Re: Using --fuzzy

on external USB drives (I have a notebook), so hard links won't work. Joe On 11/16/2014 07:38 PM, Karl O. Pinc wrote: On 11/16/2014 03:53:12 PM, Joe wrote: I have a lot of files (and directories) (up to a few hundred at a time) that I get from various sources. Some time after I get them (after

Re: Aw: Re: encrypted rsyncd - why was it never implemented?

On 12/03/2014 01:37:58 PM, Kevin Korb wrote: As far as a backup provider goes I wouldn't expect them to use rsync over SSL unless that were built into rsync in the future (and has been around long enough that most users would have it). I would expect them to either use rsync over ssh

Re: [Bug 11013] New: [patch] Mention that privileges are dropped, when use chroot is enabled in rsyncd.conf manpage

On 12/16/2014 08:45:15 PM, Kevin Korb wrote: Only root can chown. If rsync isn't running as root then it ignores the --owner part of --archive. This also makes --numeric-ids inert. Simply put, if you aren't running as root then you can only create files owned by your UID. Rsync knows

Re: Downloading a great number of files from different rsync servers for good loadbalancing and high efficiency.

On Sat, 4 Apr 2015 15:21:21 +0800 Hongyi Zhao hongyi.z...@gmail.com wrote: I'm using Debian, I want to make a local repository which can let me install packages more conveniently. Your solution will not work for mirroring debian since it does not do a 2-stage mirroring process. This is

Re: [Bug 11378] Please add a '--line-buffered' option to rsync to make logging/output more friendly with pipes/syslog/CI systems/etc.

On Sat, 04 Jul 2015 17:56:25 + samba-b...@samba.org wrote: --- Comment #2 from Nathan Neulinger nn...@neulinger.org --- Perhaps the naming is not correct on my suggested option (and I'll admit, I completely missed the outbuf option) - unfortunately, outbuf doesn't actually solve the

Re: [PATCH] Consider nanoseconds when quick-checking for unchanged files

On Wed, 20 Jan 2016 23:04:20 -0800 Wayne Davison wrote: > > The problem is that if you transfer from a filesystem that has > nanoseconds to one that does not support it, rsync would consider > most of the files to be constantly different, since the nanosecond > values would

Re: Yet another filter question

On Wed, 4 May 2016 21:09:44 -0400 Kevin Korb wrote: > That wording from the man page makes almost no sense without the > examples directly after it (and I have read it many times and know > what it is saying). Makes sense to me. The only thing I'd change is to use "in a

Re: Yet another filter question

On Wed, 4 May 2016 21:26:26 -0500 "Karl O. Pinc" <k...@meme.com> wrote: > On Wed, 4 May 2016 21:09:44 -0400 > Kevin Korb <k...@sanitarium.net> wrote: > > > That wording from the man page makes almost no sense without the > > examples directly after

Re: man page

On Wed, 27 Jul 2016 14:56:07 +0300 Marcus Fonzarelli wrote: > Therefore I'd like to suggest that the man page be changed to "which > consists of large documents and mail folders", or at least mention > another software such as LibreOffice. Submitting a patch would

Re: man page

Let me start with a reference to Nazis, just so we can reach the Godwin-point and be done with it. If somebody thinks they can improve the documentation let them submit a patch. This provides something concrete for people to comment on and the maintainers can make a decision as to whether they

Re: [Bug 12576] popt aliases allow users to bypass sudo argument restrictions

On Sun, 12 Feb 2017 02:18:35 + samba-b...@samba.org wrote: > https://bugzilla.samba.org/show_bug.cgi?id=12576 > > --- Comment #4 from Paul Donohue --- > That's an interesting solution, but it doesn't really work well for > my use case. I would like my users to be

Re: Huge directory tree: Get files to sync via tools like sysdig

On Thu, 9 Feb 2017 14:43:57 +0100 Axel Kittenberger wrote: > > > > Not only that, but inotify is not guaranteed. (At least not on > > 3.16.0. Can't say regards later versions.) So you might miss some > > changes. > > > > Got any info on that? > > I noted that MOVE_FROM

Re: Huge directory tree: Get files to sync via tools like sysdig

On Thu, 9 Feb 2017 10:55:51 +0100 Axel Kittenberger wrote: > > Has someone experience with collecting the changed files > > with a third party tool which detects which files were changed? > > I don't know of sysdig but am the developer of Lsyncd which does > exactly that,

Re: Alternatives to rsync. Was: Huge directory tree: Get files to sync via tools like sysdig

On Fri, 10 Feb 2017 11:38:34 +0100 Thomas Güttler wrote: > Yes, I think rsync is coming to the edge of its capabilities here. I > guess a different strategy is needed. > > I see these alternatives to rsync: > > - Incremental Snapshots at block-level device is

Re: How to detect hanging rsync from bash-script ?

On Mon, 10 Apr 2017 06:50:24 + (UTC) reiner otto via rsync wrote: > I am using rsync on an unreliable mobile WAN connection, which causes > rsync receiver to hang for long time, when connection is broken > during transfere. As the option "--timeout=" here does not hit

Re: Help - rsync runs from command line, fails from task scheduler, hangs at msg checking charset: UTF-8

On Fri, 28 Jul 2017 14:47:02 -0700 (PDT) leonv12 via rsync wrote: >(18 args)*msg > checking charset: UTF-8*[sender] io timeout after 3000 seconds -- > exiting[sender] _exit_cleanup(code=30, file=io.c, line=195): > enteredrsync error: timeout in data send/receive (code 30)

Re: Help - rsync runs from command line, fails from task scheduler, hangs at msg checking charset: UTF-8

On Sat, 29 Jul 2017 19:45:22 -0700 (PDT) leonv12 via rsync wrote: > I don't get why it runs from the command line but not from a > scheduled task. Any suggestions for a fix or a work-around? Talk to someone who knows about task scheduling and its failure modes? Karl

Re: [Bug 12819] [PATCH] sync() on receiving side for data consistency

On Fri, 16 Jun 2017 12:34:40 +0200 Ben RUBSON via rsync <rsync@lists.samba.org> wrote: > > On 15 Jun 2017, at 19:29, Karl O. Pinc via rsync > > <rsync@lists.samba.org> wrote: > > The problem is that the --server (and, especially, > > --daemon) documentation

Re: [Bug 12819] [PATCH] sync() on receiving side for data consistency

On Thu, 15 Jun 2017 13:23:44 + just subscribed for rsync-qa from bugzilla via rsync wrote: > https://bugzilla.samba.org/show_bug.cgi?id=12819 > > --- Comment #7 from Ben RUBSON --- > Note that my patch simply adds a sync() just after

Re: Much improved speeds of rsync via SSH - something to consider

On Mon, 2 Apr 2018 15:05:14 +0200 Thanassis Tsiodras via rsync wrote: > I recently concluded a bug hunt to trace why my rsync-ing to an SBC > was much slower than the corresponding iperf3-reported speeds. To > give a concise summary of the situation, in slow wifi links

Re: [PATCH] Optimized assembler version of md5_process() for x86-64

On Sat, 23 May 2020 10:21:31 -0700 Wayne Davison via rsync wrote: > Adding optional support for openssl's crypto library is also a good > idea. There is also libressl to consider, if you're considering libraries. Karl Free Software: "You don't pay back, you pay forward." --

Re: Enabling easier contributions to rsync

On Tue, 26 May 2020 00:43:41 +0200 uxio prego via rsync wrote: > > On 25 May 2020, at 23:55, Wayne Davison via rsync > > wrote: > > > > I've decided to give hosting it on github a try, especially since > > there's been a lot of nice contributions lately. Hopefully this > > will make it easier

Re: rsync support in authprogs - feedback requested

On Wed, 17 Feb 2021 21:52:06 -0800 Bri Hatch via rsync wrote: > I recently added initial rsync support to authprogs. > I'd be very interested in feedback For some 15 years+ (?) I've had a /root/.ssh/authorized keys line that starts with:

Re: rsync support in authprogs - feedback requested

On Thu, 18 Feb 2021 12:22:33 -0500 Kevin Korb via rsync wrote: > You should both look into rrsync. It comes with rsync and is designed > to do exactly this. I'm not really interested in restricting rsync to particular directories. That seems to be what rrsync is for, although it's a little