[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #23 from Marc Krämer  ---
@Wayne: thanks, you're right. I've overlooked this option and didn't expect it
to be harmful. Removing it really solves the issue.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #22 from Wayne Davison  ---
The --no-implied-dirs code was recreating the directories after they had been
deleted.  I changed it to avoid that if the entry it is processing is marked as
missing.  I also silenced the parent-dir warning for a missing entry that has a
parent entry that is also marked as missing.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #21 from Wayne Davison  ---
Get rid of --no-implied-dirs and --ignore-errors is also unneeded.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #20 from Wayne Davison  ---
Oops, I missed the "ls" at the end.  I'll give it a look.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #19 from Wayne Davison  ---
What in that sequence of events are you claiming is a bug?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #17 from Marc Krämer  ---
tested release 3.2.2 bug is still present:
$ cd /tmp
$ mkdir x y
$ mkdir -p x/test17/21/ff/zz
$ echo "/test17/21/ff/zz
/test17/21/ff
/test17/21
/test17
" > input.txt
$ cd y
$ rsync --no-r -ltpDcuhROHigoz --no-implied-dirs --delete
--files-from=../input.txt /tmp/x/ /tmp/y/ --delete-missing-args --ignore-errors
$ ls
test17/
$ rm -rf ../x/test17
$ rsync --no-r -ltpDcuhROHigoz --no-implied-dirs --delete
--files-from=../input.txt /tmp/x/ /tmp/y/ --delete-missing-args --ignore-errors
*deleting   test17/21/ff/
*deleting   test17/21/
*deleting   test17/
WARNING: parent dir is absent in the file list: test17
WARNING: parent dir is absent in the file list: test17/21
WARNING: parent dir is absent in the file list: test17/21/ff
rsync warning: some files vanished before they could be transferred (code 24)
at main.c(1287) [sender=3.2.2]

$ ls
test17/



so this bug is still not fixed!

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #18 from Marc Krämer  ---
tested release 3.2.2 bug is still present:
$ cd /tmp
$ mkdir x y
$ mkdir -p x/test17/21/ff/zz
$ echo "/test17/21/ff/zz
/test17/21/ff
/test17/21
/test17
" > input.txt
$ cd y
$ rsync --no-r -ltpDcuhROHigoz --no-implied-dirs --delete
--files-from=../input.txt /tmp/x/ /tmp/y/ --delete-missing-args --ignore-errors
$ ls
test17/
$ rm -rf ../x/test17
$ rsync --no-r -ltpDcuhROHigoz --no-implied-dirs --delete
--files-from=../input.txt /tmp/x/ /tmp/y/ --delete-missing-args --ignore-errors
*deleting   test17/21/ff/
*deleting   test17/21/
*deleting   test17/
WARNING: parent dir is absent in the file list: test17
WARNING: parent dir is absent in the file list: test17/21
WARNING: parent dir is absent in the file list: test17/21/ff
rsync warning: some files vanished before they could be transferred (code 24)
at main.c(1287) [sender=3.2.2]

$ ls
test17/



so this bug is still not fixed!

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #16 from Marc Krämer  ---
I'm not sure if this is still the same issue. If the directory vanishes before
transfer, e.g. issuing the command:

rsync -e ssh -d --no-r -ltpDcuhROHigoz --exclude /.sfs.conf --exclude
/.sfs.mounted --no-implied-dirs --exclude-from=/etc/sysconfig/sfs/excludes.conf
--delete-after --files-from=input.txt leonato:/mnt/bricks/www/ /tmp/x
--delete-missing-args --ignore-errors

input.txt:
/test17/21/ff/zz
/test17/21/ff
/test17/21
/test17


I get:

file has vanished: "/mnt/bricks/www/test17"
file has vanished: "/mnt/bricks/www/test17/21"
file has vanished: "/mnt/bricks/www/test17/21/ff"
*deleting   test17/21/ff/
*deleting   test17/21/
*deleting   test17/
rsync warning: some files vanished before they could be transferred (code 24)
at main.c(1637) [generator=3.1.1]


but on target the directory is still present!

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #15 from Tomas Korbar  ---
Ah. Sorry about that. I misunderstood your previous comment.
Thanks for clarification.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #14 from Wayne Davison  ---
As I mentioned above, I've already fixed the issue and left this open as a
reminder to look into an even better solution in the future.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #13 from Tomas Korbar  ---
Hi Wayne,
There is a report on this in RHEL too [0], and i reproduced it also in Fedora.
Did you make any progress?
I can help with implementation of fix if you want.
Thanks for any information you can provide.

[0] - https://bugzilla.redhat.com/show_bug.cgi?id=1727093

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

Wayne Davison  changed:

   What|Removed |Added

 Status|NEW |ASSIGNED

--- Comment #12 from Wayne Davison  ---
This check was added to try to prevent an untrusted sender from sneaking
through some bad file-list data.  However, the --delete-missing-args option
does seem to have the ability to send these incongruent lists.  I'm thinking
that the best fix is probably to have the sender fudge up the missing parent
dirs if they can be marked as missing like the missing args.  In the meantime,
I'm checking in a slightly different version of Paul's patch that makes the
receiver just complain about the missing parent dir info and continue on (it
does not return or a valid missing arg won't be deleted).

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #11 from Paul Slootman  ---
Created attachment 14775
  --> https://bugzilla.samba.org/attachment.cgi?id=14775=edit
proposed patch

This patch is based on the patch posted earlier, but checks the options
--delete-missing-args and --ignore-errors.

I see no reason why this shouldn't be patched into the main rsync sources.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 12569] Missing directory errors not ignored

[Chris Goodman via rsync]

  
  
  
?

(https://mail.edison.tech/chat-invite.html?invitorName=Chris%20Goodman=cgoody2...@gmail.com=?=572b82e3c44e452eab297ad5de8726d6=Rsync-Qa=rsync...@samba.org)
  
  

  
Sent With Edison Mail
  
  

  
  
  

  
  

  
  
>   
> On Oct 9, 2018 at 4:17 PM,   rsync (mailto:rsync@lists.samba.org)>  wrote:
>   
>   
>   
>  https://bugzilla.samba.org/show_bug.cgi?id=12569  
>
> --- Comment #10 from Marc Krämer---  
> @Axel: cool, I've played a bit with your tool, but for my needs with many  
> directories inotify was the pitfall.  
>
> I'm coauthor on sfs (https://github.com/mokraemer/sfs) which uses fuse for  
> signaling. And then, as you do, rsync for synchronization. So we have both 
> the  
> same problem here :(  
>
> --  
> You are receiving this mail because:  
> You are the QA Contact for the bug.  
>
> --  
> Please use reply-all for most replies to avoid omitting the mailing list.  
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync  
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html  
>  -- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #10 from Marc Krämer  ---
@Axel: cool, I've played a bit with your tool, but for my needs with many
directories inotify was the pitfall.

I'm coauthor on sfs (https://github.com/mokraemer/sfs) which uses fuse for
signaling. And then, as you do, rsync for synchronization. So we have both the
same problem here :(

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #9 from Axel Kittenberger  ---
@Marc, indeed. I'm the author of Lsyncd.

https://github.com/axkibe/lsyncd

If this could work properly, it would simplify things a lot, also improve
perfomance a good deal. Due to this bug I had to drop the use of --files-from
feature and keep creating include/exclude filters on the fly, which also means
rsync has to inspect way more files, which is a quite a penalty with
directories and a lot files.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #8 from Marc Krämer  ---
@Axel: you're right. This is not what we want. Even the output 
sync warning: some files vanished before they could be transferred
is not desireable if the parameter is called "ignore" there should not be any
output.
And the return code of this transfer should not indicate any errors.

I assume, you use this for automatic synchronisation between servers as well? I
don't understand why they changed the behaviour, before this change, it worked
as desired.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #7 from Axel Kittenberger  ---
No please don't close. Still not the behavior I'd expect:

"""
~$ mkdir test
~$ cd test
test$ mkdir -p src/a trg/a
test$ echo "/a/b/c" > list
test$ /usr/bin/rsync -slt --ignore-errors --force --ignore-missing-args
--delete-missing-args --files-from=list --no-implied-dirs src
localhost:`pwd`/trg
file has vanished: "/home/axel/test/src/a/b"
rsync warning: some files vanished before they could be transferred (code 24)
at main.c(1196) [sender=3.1.2]
test$ diff -r src/ trg/
Only in trg/a: b
"""

Unless I misunderstand something. It still should not create b at the target,
if it isn't in the source.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #6 from Marc Krämer  ---
ups, didn't get a notice from your reply.

Thanks for your explanation. This was not obvious to me. It should be
documented, the behaviour has changed.

You can close this one, thanks.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #5 from Dave Gordon  ---
I think you need to add "--no-implied-dirs" to get the behaviour you want.

The issue is that the contents list contains /a/b/c, so problems with that
specific file are suppressed by "--ignore-missing-args", but /a/b is not
a missing argument, it's a missing implied directory. Adding the extra flag
means that it won't be (implicitly) transferred, but would instead be created
on the destination if necessary. Then you just get a warning about the file
that's vanished:

$ rsync -slt --delete-missing-args --files-from=list src trg
file has vanished: ".../src/a/b"
ABORTING due to invalid path from sender: a/b/c
rsync error: protocol incompatibility (code 2) at generator.c(1270)
[generator=3.1.1]

$ rsync -slt --delete-missing-args --no-implied-dirs --files-from=list src trg
file has vanished: ".../src/a/b"
rsync warning: some files vanished before they could be transferred (code 24)
at main.c(1183) [sender=3.1.1]

HTH,
.Dave.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #4 from Marc Krämer  ---
My bugreport at mageia: https://bugs.mageia.org/show_bug.cgi?id=21395

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #3 from Marc Krämer  ---
that is my understanding too! And this was true before the last release.

Basic tools like rsync should not break their behaviour!

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #2 from Axel Kittenberger  ---
shouldn't "--ignore-errors" already be that parameter?
It uses --force also.

--ignore-errors --force --i-really-mean-it? :)

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12569] Missing directory errors not ignored

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #1 from Marc Krämer  ---
I'd like to point out that this change is a changed behavior that breaks some
scripts depending on this behavior.
Can you consider to change it to the original behavior, or add a new parameter,
that causes missing directories to be ignored.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html