[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #26 from Rui DeSousa  ---
(In reply to Ben RUBSON from comment #25)

That is awesome.  Thanks you for all of your efforts!

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #24 from Ben RUBSON  ---
ZFS only shares between files with dedup on.
So first rsync / diff succeeded, second gave same result as before :
# rsync -a --inplace $tmpfs/f1 $f/f3 ; echo $? ; diff $tmpfs/f1 $f/f3
0
Files /mnt/f1 and /test/f3 differ

# ls -l $f
total 1048593
-rw---  1 root  wheel  629145600  6 Mar 09:38 f2
-rw---  1 root  wheel  629145600  6 Mar 09:40 f3
# du -sm $f/*
601/test/f2
424/test/f3

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #23 from Dave Gordon  ---
Looks like this ZFS problem could be a FreeBSD-specific issue; one of the
commits mentioned in this FreeNAS bug report has the subject
  zfs_write: fix problem with writes appearing to succeed when over quota

See https://redmine.ixsystems.com/issues/26650

FreeNAS Bug #26650: Correctly report ZFS dataset quota overflows

Updated by Alexander Motin 4 months ago

Investigation of the problem brought me to FreeBSD-specific change r298105 by
avg@ on 2016-04-16. If quota overflow detected during write, the write will
fail, but the error status can be lost, falsely reporting partial completion.
As result written data are flowing to nowhere and indefinitely, as fast as CPU
can handle the loop.

HTH,
.Dave.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #22 from Dave Gordon  ---
(In reply to Ben RUBSON from comment #19)

Just to be totally certain about what ZFS may or may not share between files,
could you try this variant of your testcase:

# zfs destroy $z
# zfs create $z
# zfs set compression=off $z
# zfs set dedup=off $z
# zfs set quota=1G $z

Then with "tmpfs" referring to any non-ZFS filesystem:
# dd if=/dev/random bs=1M count=600 of=$tmpfs/f1
# rsync -a --inplace $tmpfs/f1 $f/f2 ; echo $? ; diff $tmpfs/f1 $f/f2
# dd if=/dev/random bs=1M count=600 of=$tmpfs/f1
# rsync -a --inplace $tmpfs/f1 $f/f3 ; echo $? ; diff $tmpfs/f1 $f/f3

The first rsync should succeed, and the files should be identical.
The second should fail, but if it doesn't then at least it can't be down to ZFS
finding matching datablocks.

.Dave.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #21 from Dave Gordon  ---
Created attachment 14019
  --> https://bugzilla.samba.org/attachment.cgi?id=14019=edit
Test patch to see whether fdatasync() or fsync() detects a write failure

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #20 from Dave Gordon  ---
So, looking like a ZFS issue but triggered in some way by the specific
behaviour of rsync (e.g. writing a certain block size/pattern causes the quota
error to be lost). The truss listing from a failing case should show exactly
what operations were performed on the destination filesystem; maybe someone
from the ZFS team could see whether there's some pathological case that's not
handled properly.

.Dave.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #19 from Ben RUBSON  ---
I managed to reproduce the issue on 11.0-RELEASE-p16.
Below a simple test case, without compression, without deduplication.
Note that issue is reproductible with quota, but not with userquota.

# f=/test
# z=zroot/ROOT/default$f

### quota / refquota (same results)

# zfs destroy $z
# zfs create $z
# zfs set compression=off $z
# zfs set dedup=off $z
# zfs set quota=1G $z

# dd if=/dev/random bs=1M count=600 of=$f/f1
# rsync -a --inplace $f/f1 $f/f2 ; echo $? ; diff $f/f1 $f/f2 ; rm $f/f2
0
Files /test/f1 and /test/f2 differ

# rsync -a $f/f1 $f/f2 ; echo $? ; rm $f/f2
rsync: rename "/test/.f2.6NVNwD" -> "f2": Disc quota exceeded (69)
rsync error: some files/attrs were not transferred (see previous errors)
(code 23) at main.c(1192) [sender=3.1.3]
23
// the quota error should have triggered on write instead //

### userquota

# zfs destroy $z
# zfs create $z
# zfs set compression=off $z
# zfs set dedup=off $z
# zfs set userquota@root=1G $z

# dd if=/dev/random bs=1M count=600 of=$f/f1
# rsync -a --inplace $f/f1 $f/f2 ; echo $? ; diff $f/f1 $f/f2 ; rm $f/f2
rsync: [sender] write error: Broken pipe (32)
rsync: write failed on "/test/f2": Disc quota exceeded (69)
rsync error: error in file IO (code 11) at receiver.c(404) [receiver=3.1.3]
11
Files /test/f1 and /test/f2 differ

# rsync -a $f/f1 $f/f2 ; echo $? ; rm $f/f2
rsync: [sender] write error: Broken pipe (32)
rsync: write failed on "/test/f2": Disc quota exceeded (69)
rsync error: error in file IO (code 11) at receiver.c(404) [receiver=3.1.3]
11

# dd if=/dev/random bs=1M count=535 of=$f/f1
# rsync -a --inplace $f/f1 $f/f2 ; echo $? ; diff $f/f1 $f/f2
0
# zfs get -Hp -o value used,userquota@root $z
1123188736
1073741824
# touch $f/file
touch: /test/file: Disc quota exceeded

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #18 from Rui DeSousa  ---
I also wrote a little util as well; I get the correct error in write spot.

[postgres@hades ~]$ cat 0001005E0017 | ./fwrite/fwrite
arch/0001005E0017
fwrite: write: Disc quota exceeded
[postgres@hades ~]$ echo $?
1
[postgres@hades ~]$ du -h arch/0001005E0017.GghJVR 
1.9March/0001005E0017.GghJVR


Here's the code:

#include 
#include 
#include 
#include 
#include 
#include 
#include 


#defineBUFSIZE 131072

int
main(int argc, char *argv[])
{
  int fd, r, w;
  char *buf;
  char *name;

  if (argc != 2) {
fprintf(stderr, "usage: fwrite [file]\n");
exit(1);
  }

  if ((buf = malloc(BUFSIZE)) == NULL)
err(1, "malloc");

  ++argv;
  if ((name = (char *) malloc(strlen(*argv) + 10)) == NULL)
err(1, "malloc");

  strcat(strcpy(name, *argv), ".XX");

  if ((fd = mkstemp(name)) < 0)
err(1, "mkstemp");

  while ((r = read(STDIN_FILENO, buf, BUFSIZE)) > 0)
if ((w = write(fd, buf, r)) == -1)
  err(1, "write");

  if (r < 0)
   err(1, "read");

  if (fsync(fd) != 0)
   err(1, "fsync");

  if (close(fd) != 0)
   err(1, "close");

  if (rename(name, *argv) != 0)
   err(1, "rename");

  free(name);
  exit(0);
}

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #17 from Rui DeSousa  ---
(In reply to Dave Gordon from comment #14)

Here's the output you requested.  ZFS would use the same block even if it's the
same data as don't have dedup enabled.

[postgres@hades ~]$ ls arch/
dbc1
[postgres@hades ~]$ du -h 0001005E0017 ; df -h arch/
 19M0001005E0017
Filesystem  SizeUsed   Avail Capacity  Mounted on
hydra/home/postgres/arch977M975M1.9M   100%   
/usr/home/postgres/arch
[postgres@hades ~]$ rsync -avi --inplace --debug=deltasum2,recv2,io2
0001005E0017 arch/0001005E0017
[Receiver] safe_read(0)=4
[sender] safe_read(5)=4
[sender] safe_read(5)=1
[sender] safe_read(5)=4
sending incremental file list
server_recv(2) starting pid=64788
get_local_name count=1 arch/0001005E0017
send_files mapped 0001005E0017 of size 67108864
calling match_sums 0001005E0017
>f+ 0001005E0017
sending file_sum
false_alarms=0 hash_hits=0 matches=0
recv_files(1) starting
recv_files(0001005E0017)
got file_sum
finishing 0001005E0017
[receiver] send_msg_int(100, 1)
recv_files phase=1
total: matches=0  hash_hits=0  false_alarms=0 data=67108864
recv_files phase=2
recv_files finished
[receiver] send_msg(10, 8)

sent 67,125,370 bytes  received 313 bytes  2,200,842.07 bytes/sec
total size is 67,108,864  speedup is 1.00
[postgres@hades ~]$ echo $?
0
[postgres@hades ~]$ du -h arch/0001005E0017; md5
0001005E0017 arch/0001005E0017 
1.9March/0001005E0017
MD5 (0001005E0017) = b607d345c5527152450c02c23d778cf2
MD5 (arch/0001005E0017) = f90772e440b04b63490a88e7dafeac84
[postgres@hades ~]$ touch 0001005E0017 
[postgres@hades ~]$ rsync -avic --inplace --debug=deltasum3,recv2,io2
0001005E0017 arch/0001005E0017
[Receiver] safe_read(0)=4
[sender] safe_read(5)=4
[sender] safe_read(5)=1
[sender] safe_read(5)=4
sending incremental file list
server_recv(2) starting pid=67722
get_local_name count=1 arch/0001005E0017
count=0 n=0 rem=0
send_files mapped 0001005E0017 of size 67108864
calling match_sums 0001005E0017
>fc.t.. 0001005E0017
recv_files(1) starting
recv_files(0001005E0017)
recv mapped 0001005E0017 of size 67108864
data recv 32768 at 0
data recv 32768 at 32768
data recv 32768 at 65536
data recv 32768 at 98304
data recv 32768 at 131072
data recv 32768 at 163840
data recv 32768 at 196608
data recv 32768 at 229376
data recv 32768 at 262144
data recv 32768 at 294912
data recv 32768 at 327680
data recv 32768 at 360448
data recv 32768 at 393216
data recv 32768 at 425984
data recv 32768 at 458752
data recv 32768 at 491520
data recv 32768 at 524288
data recv 32768 at 557056
data recv 32768 at 589824
data recv 32768 at 622592
data recv 32768 at 655360
data recv 32768 at 688128
data recv 32768 at 720896
data recv 32768 at 753664
data recv 32768 at 786432
data recv 32768 at 819200
data recv 32768 at 851968
data recv 32768 at 884736
data recv 32768 at 917504
data recv 32768 at 950272
data recv 32768 at 983040
data recv 32768 at 1015808
data recv 32768 at 1048576
data recv 32768 at 1081344
data recv 32768 at 1114112
data recv 32768 at 1146880
data recv 32768 at 1179648
data recv 32768 at 1212416
data recv 32768 at 1245184
data recv 32768 at 1277952
data recv 32768 at 1310720
data recv 32768 at 1343488
data recv 32768 at 1376256
data recv 32768 at 1409024
data recv 32768 at 1441792
data recv 32768 at 1474560
data recv 32768 at 1507328
data recv 32768 at 1540096
data recv 32768 at 1572864
data recv 32768 at 1605632
data recv 32768 at 1638400
data recv 32768 at 1671168
data recv 32768 at 1703936
data recv 32768 at 1736704
data recv 32768 at 1769472
data recv 32768 at 1802240
data recv 32768 at 1835008
data recv 32768 at 1867776
data recv 32768 at 1900544
data recv 32768 at 1933312
data recv 32768 at 1966080
data recv 32768 at 1998848
data recv 32768 at 2031616
data recv 32768 at 2064384
data recv 32768 at 2097152
data recv 32768 at 2129920
data recv 32768 at 2162688
data recv 32768 at 2195456
data recv 32768 at 2228224
data recv 32768 at 2260992
data recv 32768 at 2293760
data recv 32768 at 2326528
data recv 32768 at 2359296
data recv 32768 at 2392064
data recv 32768 at 2424832
data recv 32768 at 2457600
data recv 32768 at 2490368
data recv 32768 at 2523136
data recv 32768 at 2555904
data recv 32768 at 2588672
data recv 32768 at 2621440
data recv 32768 at 2654208
data recv 32768 at 2686976
data recv 32768 at 2719744
data recv 32768 at 2752512
data recv 32768 at 2785280
data recv 32768 at 2818048
data recv 32768 at 2850816
data recv 32768 at 2883584
data recv 32768 at 2916352
data recv 32768 at 2949120
data recv 32768 at 2981888
data recv 32768 at 3014656
data recv 32768 at 3047424
data recv 32768 at 

[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #16 from Rui DeSousa  ---
(In reply to Ben RUBSON from comment #15)

I just set the quota property.

NAME  PROPERTY   VALUE SOURCE
hydra/home/postgres/arch  quota  1Glocal
hydra/home/postgres/arch  reservationnone  default
hydra/home/postgres/arch  refquota   none  default
hydra/home/postgres/arch  compressionlz4   inherited from
hydra/home/postgres
hydra/home/postgres/arch  compressratio  3.66x -

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #15 from Ben RUBSON  ---
Rui just to be sure, which type of ZFS quota are you using ?
quota ? refquota ? userquota ?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 13317] rsync returns success when target filesystem is full

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=13317

--- Comment #14 from Dave Gordon  ---
(In reply to Rui DeSousa from comment #6)

So you now have an example which reliably produces a bad outcome (corrupted
file)? With the combination of
(a) insufficient space before copying, and
(b) --inplace so that no rename is needed
you get no error from rsync but the file uses far less space than it should?
And the contents are not as expected?

Perhaps you could try this sequence:
$ src=0001005E0017
$ dst=arch/0001005E0017
$ rm $dst
$ rsync -avi --inplace --debug=deltasum2,recv2,io2 $src $dst
$ touch $src
$ rsync -avic --inplace --debug=deltasum3,recv2,io2 $src $dst
$ md5sum $src $dst
and if the md5sum shows a discrepancy then also capture a truss listing of all
the syscalls made by the first run of rsync (where the destination does not
exist).

Rationale:
I'm wondering whether ZFS if able to detect that the destination file is
identical to the source file (even though it's on a different logical
filesystem, it's probably still in the same pool) and merge the disk blocks
(retrospectively?)

So we start with a file whose logical size is 64M, but which is compressed to
19M on disk. ZFS uncompresses it on the fly and delivers 64M of data to the
first rsync. rsync sequentially writes 64M, checking the success of each write.
The last write should end at an offset of 64M, then the destination file is
closed (and the return from that is checked). The truss listing will show how
many writes were made, and whether any of them failed. ZFS will recompress (and
dedup?) the written data, resulting in an unknown amount of new space being
used.

The second rsync will read both the source and destination files -- presumably
both now being decompressed on demand -- and compare checksums. Any mismatch
here will result in (parts of) the file being retransferred; obviously, this
shouldn't happen if the first transfer succeeded.

md5sum will check whether the files really do match.

HTH,
.Dave.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html