[Touch-packages] [Bug 1659950] Re: rsync -u --inplace --partial -a can't resume transfer
** Changed in: rsync Status: Unknown => New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1659950 Title: rsync -u --inplace --partial -a can't resume transfer Status in rsync: New Status in rsync package in Ubuntu: Opinion Bug description: Suppose you have a file in hostA: hostA$ ls -l /tmp/files -rw-rw-r-- 2 root root 563016 Jan 10 15:01 test.txt You download it from the hostB using: hostB$ rsync -u --inplace --partial -a hostA::files/* . If the transfer is aborted, hostB will get only a partial file: hostB$ ls -l /tmp/files -rw-rw-r-- 2 root root 2024 Jan 11 18:00 test.txt BUT the ctime/mtime of hostB/test.txt now is NEWER than hostA/test.txt(and mtime == ctime). So, if you run the same rsync -u command again: hostB$ rsync -u --inplace --partial -a hostA::files/* . Rsync will SKIP THE FILE, because hostB/test.txt is "newer" than hostA/test.txt. So you CAN'T resume using rsync -u command, and you will think there are no differences. To avoid this bug, rsync must create the file with ctime=mtime=0. And if the file already exists before transfer, rsync -u must not change his current ctime/mtime. Ctime/mtime must be updated ONLY after the transfer was successfully completed. Note this is really need because there are scenarios where checksum comparison can't be used, only comparison by time. For example, to avoid deleting changes made in hostB to test.txt. Also I need to use --inplace. To manage notifications about this bug go to: https://bugs.launchpad.net/rsync/+bug/1659950/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1659950] Re: rsync -u --inplace --partial -a can't resume transfer
Thanks Nahuel! We're doing some retriage and cleanup of old bugs, and I
stumbled on this one :-)
I linked your upstream bug report to this bug, Launchpad will keep its
status in sync. You may have noticed that I didn't set the status of
this bug to Wontfix but to Opinion ("Doesn't fit with the project, but
can be discussed"), I think it fits better.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to rsync in Ubuntu.
https://bugs.launchpad.net/bugs/1659950
Title:
rsync -u --inplace --partial -a can't resume transfer
Status in rsync:
Unknown
Status in rsync package in Ubuntu:
Opinion
Bug description:
Suppose you have a file in hostA:
hostA$ ls -l /tmp/files
-rw-rw-r-- 2 root root 563016 Jan 10 15:01 test.txt
You download it from the hostB using:
hostB$ rsync -u --inplace --partial -a hostA::files/* .
If the transfer is aborted, hostB will get only a partial file:
hostB$ ls -l /tmp/files
-rw-rw-r-- 2 root root 2024 Jan 11 18:00 test.txt
BUT the ctime/mtime of hostB/test.txt now is NEWER than hostA/test.txt(and
mtime == ctime). So, if you run the same rsync -u command again:
hostB$ rsync -u --inplace --partial -a hostA::files/* .
Rsync will SKIP THE FILE, because hostB/test.txt is "newer" than
hostA/test.txt. So you CAN'T resume using rsync -u command, and you
will think there are no differences.
To avoid this bug, rsync must create the file with ctime=mtime=0. And
if the file already exists before transfer, rsync -u must not change
his current ctime/mtime. Ctime/mtime must be updated ONLY after the
transfer was successfully completed.
Note this is really need because there are scenarios where checksum
comparison can't be used, only comparison by time. For example, to
avoid deleting changes made in hostB to test.txt. Also I need to use
--inplace.
To manage notifications about this bug go to:
https://bugs.launchpad.net/rsync/+bug/1659950/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1659950] Re: rsync -u --inplace --partial -a can't resume transfer
** Also affects: rsync via https://github.com/WayneD/rsync/issues/236 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1659950 Title: rsync -u --inplace --partial -a can't resume transfer Status in rsync: Unknown Status in rsync package in Ubuntu: Opinion Bug description: Suppose you have a file in hostA: hostA$ ls -l /tmp/files -rw-rw-r-- 2 root root 563016 Jan 10 15:01 test.txt You download it from the hostB using: hostB$ rsync -u --inplace --partial -a hostA::files/* . If the transfer is aborted, hostB will get only a partial file: hostB$ ls -l /tmp/files -rw-rw-r-- 2 root root 2024 Jan 11 18:00 test.txt BUT the ctime/mtime of hostB/test.txt now is NEWER than hostA/test.txt(and mtime == ctime). So, if you run the same rsync -u command again: hostB$ rsync -u --inplace --partial -a hostA::files/* . Rsync will SKIP THE FILE, because hostB/test.txt is "newer" than hostA/test.txt. So you CAN'T resume using rsync -u command, and you will think there are no differences. To avoid this bug, rsync must create the file with ctime=mtime=0. And if the file already exists before transfer, rsync -u must not change his current ctime/mtime. Ctime/mtime must be updated ONLY after the transfer was successfully completed. Note this is really need because there are scenarios where checksum comparison can't be used, only comparison by time. For example, to avoid deleting changes made in hostB to test.txt. Also I need to use --inplace. To manage notifications about this bug go to: https://bugs.launchpad.net/rsync/+bug/1659950/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1659950] Re: rsync -u --inplace --partial -a can't resume transfer
Reported to upstream at: https://github.com/WayneD/rsync/issues/236 ** Bug watch added: github.com/WayneD/rsync/issues #236 https://github.com/WayneD/rsync/issues/236 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1659950 Title: rsync -u --inplace --partial -a can't resume transfer Status in rsync package in Ubuntu: Opinion Bug description: Suppose you have a file in hostA: hostA$ ls -l /tmp/files -rw-rw-r-- 2 root root 563016 Jan 10 15:01 test.txt You download it from the hostB using: hostB$ rsync -u --inplace --partial -a hostA::files/* . If the transfer is aborted, hostB will get only a partial file: hostB$ ls -l /tmp/files -rw-rw-r-- 2 root root 2024 Jan 11 18:00 test.txt BUT the ctime/mtime of hostB/test.txt now is NEWER than hostA/test.txt(and mtime == ctime). So, if you run the same rsync -u command again: hostB$ rsync -u --inplace --partial -a hostA::files/* . Rsync will SKIP THE FILE, because hostB/test.txt is "newer" than hostA/test.txt. So you CAN'T resume using rsync -u command, and you will think there are no differences. To avoid this bug, rsync must create the file with ctime=mtime=0. And if the file already exists before transfer, rsync -u must not change his current ctime/mtime. Ctime/mtime must be updated ONLY after the transfer was successfully completed. Note this is really need because there are scenarios where checksum comparison can't be used, only comparison by time. For example, to avoid deleting changes made in hostB to test.txt. Also I need to use --inplace. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1659950/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1659950] Re: rsync -u --inplace --partial -a can't resume transfer
Hello Paride, 4 years have passed from my initial report, I forgot completely about this bug :) Fine, I will post this issue to upstream. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1659950 Title: rsync -u --inplace --partial -a can't resume transfer Status in rsync package in Ubuntu: Opinion Bug description: Suppose you have a file in hostA: hostA$ ls -l /tmp/files -rw-rw-r-- 2 root root 563016 Jan 10 15:01 test.txt You download it from the hostB using: hostB$ rsync -u --inplace --partial -a hostA::files/* . If the transfer is aborted, hostB will get only a partial file: hostB$ ls -l /tmp/files -rw-rw-r-- 2 root root 2024 Jan 11 18:00 test.txt BUT the ctime/mtime of hostB/test.txt now is NEWER than hostA/test.txt(and mtime == ctime). So, if you run the same rsync -u command again: hostB$ rsync -u --inplace --partial -a hostA::files/* . Rsync will SKIP THE FILE, because hostB/test.txt is "newer" than hostA/test.txt. So you CAN'T resume using rsync -u command, and you will think there are no differences. To avoid this bug, rsync must create the file with ctime=mtime=0. And if the file already exists before transfer, rsync -u must not change his current ctime/mtime. Ctime/mtime must be updated ONLY after the transfer was successfully completed. Note this is really need because there are scenarios where checksum comparison can't be used, only comparison by time. For example, to avoid deleting changes made in hostB to test.txt. Also I need to use --inplace. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1659950/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1659950] Re: rsync -u --inplace --partial -a can't resume transfer
Hello Nahuel, According to your reasoning I assume the "root" issue should still affect the versions of rsync shipped with the newer Ubuntu releases. I agree it would be nice for the behavior you describe to be documented, but I don't think it's worth patching the Ubuntu package for it. I recognize we have a papercut here, but I don't think the fix for it belongs to Ubuntu; it should instead be driven (and thus also validated) upstream. For this reason I'm marking this bug as a Won't Fix. Should you disagree with my assessment please comment back and set the bug status back to New, we'll look at it again. Thanks! ** Changed in: rsync (Ubuntu) Status: Triaged => Opinion -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1659950 Title: rsync -u --inplace --partial -a can't resume transfer Status in rsync package in Ubuntu: Opinion Bug description: Suppose you have a file in hostA: hostA$ ls -l /tmp/files -rw-rw-r-- 2 root root 563016 Jan 10 15:01 test.txt You download it from the hostB using: hostB$ rsync -u --inplace --partial -a hostA::files/* . If the transfer is aborted, hostB will get only a partial file: hostB$ ls -l /tmp/files -rw-rw-r-- 2 root root 2024 Jan 11 18:00 test.txt BUT the ctime/mtime of hostB/test.txt now is NEWER than hostA/test.txt(and mtime == ctime). So, if you run the same rsync -u command again: hostB$ rsync -u --inplace --partial -a hostA::files/* . Rsync will SKIP THE FILE, because hostB/test.txt is "newer" than hostA/test.txt. So you CAN'T resume using rsync -u command, and you will think there are no differences. To avoid this bug, rsync must create the file with ctime=mtime=0. And if the file already exists before transfer, rsync -u must not change his current ctime/mtime. Ctime/mtime must be updated ONLY after the transfer was successfully completed. Note this is really need because there are scenarios where checksum comparison can't be used, only comparison by time. For example, to avoid deleting changes made in hostB to test.txt. Also I need to use --inplace. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1659950/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1659950] Re: rsync -u --inplace --partial -a can't resume transfer
** Changed in: rsync (Ubuntu) Status: New => Triaged ** Changed in: rsync (Ubuntu) Importance: Undecided => Low -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1659950 Title: rsync -u --inplace --partial -a can't resume transfer Status in rsync package in Ubuntu: Triaged Bug description: Suppose you have a file in hostA: hostA$ ls -l /tmp/files -rw-rw-r-- 2 root root 563016 Jan 10 15:01 test.txt You download it from the hostB using: hostB$ rsync -u --inplace --partial -a hostA::files/* . If the transfer is aborted, hostB will get only a partial file: hostB$ ls -l /tmp/files -rw-rw-r-- 2 root root 2024 Jan 11 18:00 test.txt BUT the ctime/mtime of hostB/test.txt now is NEWER than hostA/test.txt(and mtime == ctime). So, if you run the same rsync -u command again: hostB$ rsync -u --inplace --partial -a hostA::files/* . Rsync will SKIP THE FILE, because hostB/test.txt is "newer" than hostA/test.txt. So you CAN'T resume using rsync -u command, and you will think there are no differences. To avoid this bug, rsync must create the file with ctime=mtime=0. And if the file already exists before transfer, rsync -u must not change his current ctime/mtime. Ctime/mtime must be updated ONLY after the transfer was successfully completed. Note this is really need because there are scenarios where checksum comparison can't be used, only comparison by time. For example, to avoid deleting changes made in hostB to test.txt. Also I need to use --inplace. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1659950/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1659950] Re: rsync -u --inplace --partial -a can't resume transfer
Researching more, I see POSIX has NO way to disable mtime updating while calling write()'s, so there is no way to atomically leave a partial file with an mtime=0 mark while using --inplace. "rsync --update --partial" (no --inplace) can do it because it transfers first to a temporal file, then updates his mtime (to 0 if unsuccessful, or to the original file mtime if transfer completed), and then rename()'s the file. So now I think this can't be solved, but a warning should be placed in the rsync manpage about this "-u --partial" unexpected behavior. Take this as a documentation bug. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1659950 Title: rsync -u --inplace --partial -a can't resume transfer Status in rsync package in Ubuntu: New Bug description: Suppose you have a file in hostA: hostA$ ls -l /tmp/files -rw-rw-r-- 2 root root 563016 Jan 10 15:01 test.txt You download it from the hostB using: hostB$ rsync -u --inplace --partial -a hostA::files/* . If the transfer is aborted, hostB will get only a partial file: hostB$ ls -l /tmp/files -rw-rw-r-- 2 root root 2024 Jan 11 18:00 test.txt BUT the ctime/mtime of hostB/test.txt now is NEWER than hostA/test.txt(and mtime == ctime). So, if you run the same rsync -u command again: hostB$ rsync -u --inplace --partial -a hostA::files/* . Rsync will SKIP THE FILE, because hostB/test.txt is "newer" than hostA/test.txt. So you CAN'T resume using rsync -u command, and you will think there are no differences. To avoid this bug, rsync must create the file with ctime=mtime=0. And if the file already exists before transfer, rsync -u must not change his current ctime/mtime. Ctime/mtime must be updated ONLY after the transfer was successfully completed. Note this is really need because there are scenarios where checksum comparison can't be used, only comparison by time. For example, to avoid deleting changes made in hostB to test.txt. Also I need to use --inplace. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1659950/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1659950] Re: rsync -u --inplace --partial -a can't resume transfer
rsync version: rsync version 3.1.1 protocol version 31 ubuntu version: 3.1.1-3ubuntu1 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1659950 Title: rsync -u --inplace --partial -a can't resume transfer Status in rsync package in Ubuntu: New Bug description: Suppose you have a file in hostA: hostA$ ls -l /tmp/files -rw-rw-r-- 2 root root 563016 Jan 10 15:01 test.txt You download it from the hostB using: hostB$ rsync -u --inplace --partial -a hostA::files/* . If the transfer is aborted, hostB will get only a partial file: hostB$ ls -l /tmp/files -rw-rw-r-- 2 root root 2024 Jan 11 18:00 test.txt BUT the ctime/mtime of hostB/test.txt now is NEWER than hostA/test.txt(and mtime == ctime). So, if you run the same rsync -u command again: hostB$ rsync -u --inplace --partial -a hostA::files/* . Rsync will SKIP THE FILE, because hostB/test.txt is "newer" than hostA/test.txt. So you CAN'T resume using rsync -u command, and you will think there are no differences. To avoid this bug, rsync must create the file with ctime=mtime=0. And if the file already exists before transfer, rsync -u must not change his current ctime/mtime. Ctime/mtime must be updated ONLY after the transfer was successfully completed. Note this is really need because there are scenarios where checksum comparison can't be used, only comparison by time. For example, to avoid deleting changes made in hostB to test.txt. Also I need to use --inplace. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1659950/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1659950] Re: rsync -u --inplace --partial -a can't resume transfer
A reproducible test: ~$ cd /tmp/ /tmp$ mkdir a b /tmp$ cd a /tmp/a$ head -c 10 /dev/urandom > test /tmp/a$ ls -l test -rw-r--r-- 1 nahuel nahuel 10 Jan 27 18:27 test /tmp/a$ cd ../b /tmp/b$ timeout 3 rsync -u --inplace --partial --bwlimit=2k --progress -va ../a/test . sending incremental file list test 32,768 32%0.00kB/s0:00:00 rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(504) [generator=3.1.1] /tmp/b$ ls -l test -rw--- 1 nahuel nahuel 0 Jan 27 18:28 test /tmp/b$ timeout 3 rsync -u --inplace --partial --bwlimit=2k --progress -va ../a/test . sending incremental file list sent 59 bytes received 12 bytes 142.00 bytes/sec total size is 100,000 speedup is 1,408.45 /tmp/b$ ls -l test -rw--- 1 nahuel nahuel 0 Jan 27 18:28 test nahuel@serge:/tmp/b$ -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsync in Ubuntu. https://bugs.launchpad.net/bugs/1659950 Title: rsync -u --inplace --partial -a can't resume transfer Status in rsync package in Ubuntu: New Bug description: Suppose you have a file in hostA: hostA$ ls -l /tmp/files -rw-rw-r-- 2 root root 563016 Jan 10 15:01 test.txt You download it from the hostB using: hostB$ rsync -u --inplace --partial -a hostA::files/* . If the transfer is aborted, hostB will get only a partial file: hostB$ ls -l /tmp/files -rw-rw-r-- 2 root root 2024 Jan 11 18:00 test.txt BUT the ctime/mtime of hostB/test.txt now is NEWER than hostA/test.txt(and mtime == ctime). So, if you run the same rsync -u command again: hostB$ rsync -u --inplace --partial -a hostA::files/* . Rsync will SKIP THE FILE, because hostB/test.txt is "newer" than hostA/test.txt. So you CAN'T resume using rsync -u command, and you will think there are no differences. To avoid this bug, rsync must create the file with ctime=mtime=0. And if the file already exists before transfer, rsync -u must not change his current ctime/mtime. Ctime/mtime must be updated ONLY after the transfer was successfully completed. Note this is really need because there are scenarios where checksum comparison can't be used, only comparison by time. For example, to avoid deleting changes made in hostB to test.txt. Also I need to use --inplace. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1659950/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
