Re: [Bug 12732] hard links can cause rsync to block or to silently skip files

2017-04-05 Thread L. A. Walsh via rsync

Kevin Korb via rsync wrote:

Note that in all --*-dest options if the path is relative it is relative
to the target dir not the "$PWD".  I like to always use absolute paths
because of this.  But essentially, the command with the instances of
$PWD vs without them the paths aren't the same.  If you want all
relative to . paths it would be:
rsync -aHvv --compare-dest=../dstlt/. srclt/. baklt/.
  

---
   Good point -- though it still doesn't explain the differing
output when dstlt or baklt are on different file systems (one
works, while the other was worse)


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: [Bug 12732] hard links can cause rsync to block or to silently skip files

2017-04-05 Thread Kevin Korb via rsync
Note that in all --*-dest options if the path is relative it is relative
to the target dir not the "$PWD".  I like to always use absolute paths
because of this.  But essentially, the command with the instances of
$PWD vs without them the paths aren't the same.  If you want all
relative to . paths it would be:
rsync -aHvv --compare-dest=../dstlt/. srclt/. baklt/.

Also, all these params that end with /. the . is kinda pointless despite
being points ;)

On 04/05/2017 07:19 PM, L A Walsh via rsync wrote:
> just subscribed for rsync-qa from bugzilla via rsync wrote:
>> Probably using different options? Can this be some sort of Heisenbug,
>> nobody can reproduce? Do the two sequences of shell commands work for
>> you as expected? Please note that both rsync commands in the mail
>> generated by bugzilla are split into two lines (each): Both rsync
>> commands should read
>>   
> ---
> Actually, I am reproducing your bug exactly as you entered
> it.
> And it is very sensitive to options.
> 
> When I take out the $PWD it works.
> 
> First your case:
> /tmp> rsync -aHvv --compare-dest=$PWD/dstlt/. $PWD/srclt/. $PWD/baklt/.
> |& tee testlt.log
> sending incremental file list
> delta-transmission disabled for local transfer or --whole-file
> ./
> a is uptodate
> d is uptodate
> b
> a => b
> /tmp> ll -i srclt dstlt baklt
> baklt:
> total 8
> 17305770 -rw-rw-r-- 2 2 Apr  5 15:46 a
> 17305770 -rw-rw-r-- 2 2 Apr  5 15:46 b
> dstlt:
> total 8
> 8474888 -rw-rw-r-- 1 2 Apr  5 15:46 a
> 8474896 -rw-rw-r-- 1 2 Apr  5 15:46 d
> srclt:
> total 16
> 8866 -rw-rw-r-- 2 2 Apr  5 15:46 a
> 8866 -rw-rw-r-- 2 2 Apr  5 15:46 b
> 8867 -rw-rw-r-- 2 2 Apr  5 15:46 c
> 8867 -rw-rw-r-- 2 2 Apr  5 15:46 d
> 
> Then w/o $PWD:
> /tmp> rsync -aHvv --compare-dest=dstlt/. srclt/. baklt/. |& tee testlt.log
> sending incremental file list
> --compare-dest arg does not exist: dstlt/.
> delta-transmission disabled for local transfer or --whole-file
> ./
> b
> d
> a => b
> c => d
> /tmp> ll -i srclt dstlt baklt
> baklt:
> total 16
> 17305770 -rw-rw-r-- 2 2 Apr  5 15:46 a
> 17305770 -rw-rw-r-- 2 2 Apr  5 15:46 b
> 17305782 -rw-rw-r-- 2 2 Apr  5 15:46 c
> 17305782 -rw-rw-r-- 2 2 Apr  5 15:46 d
> dstlt:
> total 8
> 8474888 -rw-rw-r-- 1 2 Apr  5 15:46 a
> 8474896 -rw-rw-r-- 1 2 Apr  5 15:46 d
> srclt:
> total 16
> 8866 -rw-rw-r-- 2 2 Apr  5 15:46 a
> 8866 -rw-rw-r-- 2 2 Apr  5 15:46 b
> 8867 -rw-rw-r-- 2 2 Apr  5 15:46 c
> 8867 -rw-rw-r-- 2 2 Apr  5 15:46 d
> --
> 
> However, if "dstl" is a different file system, it works:
>>  rsync -aHv --compare-dest=~/dstlt/. $PWD/srclt/. $PWD/baklt/. |& tee 
> testlt.log  sending incremental file list
> --compare-dest arg does not exist: ~/dstlt/.
> ./
> b
> d
> a => b
> c => d
> 
> In my usage both the compare-dest & baklt were 'technically'
> different file systems (though the compare-dest was a snap
> shot).
> ---
> ARG: if baklt is on a different FS, but src+dst are same:
> /tmp> rsync -aHv --compare-dest=$PWD/dstlt/. $PWD/srclt/. ~/baklt/. |&
> tee testlt.log  sending incremental file list
> ./
> b
> a => b
> /tmp> ll -i srclt dstlt ~/baklt
> /home/law/baklt:
> total 8
> 3677653445 -rw-rw-r-- 2 2 Apr  5 15:46 a
> 3677653445 -rw-rw-r-- 2 2 Apr  5 15:46 b
> 
> dstlt:
> total 8
> 8474888 -rw-rw-r-- 1 2 Apr  5 15:46 a
> 8474896 -rw-rw-r-- 1 2 Apr  5 15:46 d
> 
> srclt:
> total 16
> 8866 -rw-rw-r-- 2 2 Apr  5 15:46 a
> 8866 -rw-rw-r-- 2 2 Apr  5 15:46 b
> 8867 -rw-rw-r-- 2 2 Apr  5 15:46 c
> 8867 -rw-rw-r-- 2 2 Apr  5 15:46 d
> 
> 
> Ok... something's screwy here.  But no hangs.
> 
> Like I said in my 1st response, if some hard links
> were missing in the bak directory, I really would be
> unlikely to notice, as I didn't even try to check
> them (way too many files), but I never got a hang --
> and that's the part I'm thinking might be nfs
> related, since I've seen several issues with nfs not
> working the same way as a local fs.
> 
> I usually use smbfs -- in my usage, it's faster and I
> usually have fewer compatibility problems.
> (faster meaning ~ Reads~162MB/s, W~220MB/s,
> though i've seen explorer hit over 400MB/s).
> (using a 10Gb link).
> 
> But the above testing shows some unexplained behaviors
> out of rsync that sure look like a bug.
> 
> Good test case!
> :-)
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.

Re: [Bug 12732] hard links can cause rsync to block or to silently skip files

2017-04-05 Thread L A Walsh via rsync

just subscribed for rsync-qa from bugzilla via rsync wrote:

Probably using different options? Can this be some sort of Heisenbug,
nobody can reproduce? Do the two sequences of shell commands work for
you as expected? Please note that both rsync commands in the mail
generated by bugzilla are split into two lines (each): Both rsync
commands should read
  

---
Actually, I am reproducing your bug exactly as you entered
it. 


And it is very sensitive to options.

When I take out the $PWD it works.

First your case:
/tmp> rsync -aHvv --compare-dest=$PWD/dstlt/. $PWD/srclt/. $PWD/baklt/. 
|& tee testlt.log

sending incremental file list
delta-transmission disabled for local transfer or --whole-file
./
a is uptodate
d is uptodate
b
a => b
/tmp> ll -i srclt dstlt baklt
baklt:
total 8
17305770 -rw-rw-r-- 2 2 Apr  5 15:46 a
17305770 -rw-rw-r-- 2 2 Apr  5 15:46 b
dstlt:
total 8
8474888 -rw-rw-r-- 1 2 Apr  5 15:46 a
8474896 -rw-rw-r-- 1 2 Apr  5 15:46 d
srclt:
total 16
8866 -rw-rw-r-- 2 2 Apr  5 15:46 a
8866 -rw-rw-r-- 2 2 Apr  5 15:46 b
8867 -rw-rw-r-- 2 2 Apr  5 15:46 c
8867 -rw-rw-r-- 2 2 Apr  5 15:46 d

Then w/o $PWD:
/tmp> rsync -aHvv --compare-dest=dstlt/. srclt/. baklt/. |& tee testlt.log
sending incremental file list
--compare-dest arg does not exist: dstlt/.
delta-transmission disabled for local transfer or --whole-file
./
b
d
a => b
c => d
/tmp> ll -i srclt dstlt baklt
baklt:
total 16
17305770 -rw-rw-r-- 2 2 Apr  5 15:46 a
17305770 -rw-rw-r-- 2 2 Apr  5 15:46 b
17305782 -rw-rw-r-- 2 2 Apr  5 15:46 c
17305782 -rw-rw-r-- 2 2 Apr  5 15:46 d
dstlt:
total 8
8474888 -rw-rw-r-- 1 2 Apr  5 15:46 a
8474896 -rw-rw-r-- 1 2 Apr  5 15:46 d
srclt:
total 16
8866 -rw-rw-r-- 2 2 Apr  5 15:46 a
8866 -rw-rw-r-- 2 2 Apr  5 15:46 b
8867 -rw-rw-r-- 2 2 Apr  5 15:46 c
8867 -rw-rw-r-- 2 2 Apr  5 15:46 d
--

However, if "dstl" is a different file system, it works:
 rsync -aHv --compare-dest=~/dstlt/. $PWD/srclt/. $PWD/baklt/. |& tee 
testlt.log  
sending incremental file list

--compare-dest arg does not exist: ~/dstlt/.
./
b
d
a => b
c => d

In my usage both the compare-dest & baklt were 'technically'
different file systems (though the compare-dest was a snap
shot).
---
ARG: if baklt is on a different FS, but src+dst are same:
/tmp> rsync -aHv --compare-dest=$PWD/dstlt/. $PWD/srclt/. ~/baklt/. |& 
tee testlt.log  
sending incremental file list

./
b
a => b
/tmp> ll -i srclt dstlt ~/baklt
/home/law/baklt:
total 8
3677653445 -rw-rw-r-- 2 2 Apr  5 15:46 a
3677653445 -rw-rw-r-- 2 2 Apr  5 15:46 b

dstlt:
total 8
8474888 -rw-rw-r-- 1 2 Apr  5 15:46 a
8474896 -rw-rw-r-- 1 2 Apr  5 15:46 d

srclt:
total 16
8866 -rw-rw-r-- 2 2 Apr  5 15:46 a
8866 -rw-rw-r-- 2 2 Apr  5 15:46 b
8867 -rw-rw-r-- 2 2 Apr  5 15:46 c
8867 -rw-rw-r-- 2 2 Apr  5 15:46 d


Ok... something's screwy here.  But no hangs.

Like I said in my 1st response, if some hard links
were missing in the bak directory, I really would be
unlikely to notice, as I didn't even try to check
them (way too many files), but I never got a hang --
and that's the part I'm thinking might be nfs
related, since I've seen several issues with nfs not
working the same way as a local fs.

I usually use smbfs -- in my usage, it's faster and I
usually have fewer compatibility problems.
(faster meaning ~ Reads~162MB/s, W~220MB/s,
though i've seen explorer hit over 400MB/s).
(using a 10Gb link).

But the above testing shows some unexplained behaviors
out of rsync that sure look like a bug.

Good test case!
:-)










--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html