Re: [gentoo-user] is "scp" reliable?

[J. Roeleveld]

On Tuesday, June 1, 2021 6:21:52 PM CEST n952162 wrote:
> On 6/1/21 6:42 AM, J. Roeleveld wrote:
> > If there are differences, I would definitely suspect memory and CPU.
> > 
> > --
> > Joost
> 
> CPU?  USB was mentioned which set off alarm bells for me.  In general
> though, I would suspect the media  - either source of destination.

The issue was first seen using SCP-copies. Eg, over network.
USB was only mentioned as being used as an alternative, which also didn't 
work.

--
Joost

Re: [gentoo-user] is "scp" reliable?

[n952162]

On 6/1/21 6:42 AM, J. Roeleveld wrote:


If there are differences, I would definitely suspect memory and CPU.

--
Joost





CPU?  USB was mentioned which set off alarm bells for me.  In general
though, I would suspect the media  - either source of destination.

Re: [gentoo-user] is "scp" reliable?

[Adam Carter]

On Tue, Jun 1, 2021 at 2:46 PM J. Roeleveld  wrote:

> On Saturday, May 29, 2021 11:04:44 PM CEST Mark Knecht wrote:
> > On Sat, May 29, 2021 at 1:33 PM  wrote:
> > 
> >
> > > Another mystery.
> > > I copied the file to USB 1TB sandisk.
> > > md5sum check OK same as my computer
> > >
> > >
>
> > Different revisions of md5sum possibly?
>
> I have never had issues with different md5sum tools.
>

Yes - it hard to imagine a bug of that seriousness not being detected.
Getting it right is the tool's raison d'etre.

Re: [gentoo-user] is "scp" reliable?

[J. Roeleveld]

On Saturday, May 29, 2021 11:04:44 PM CEST Mark Knecht wrote:
> On Sat, May 29, 2021 at 1:33 PM  wrote:
> 
> 
> > Another mystery.
> > I copied the file to USB 1TB sandisk.
> > md5sum check OK same as my computer
> > 
> > 

> Different revisions of md5sum possibly?

I have never had issues with different md5sum tools.
I often use md5sum along with sha1sum to check file-integrity of downloaded 
files. The checksums being provided by the source.

If there are differences, I would definitely suspect memory and CPU.

--
Joost

Re: [SOLVED] [gentoo-user] is "scp" reliable?

[thelma]

On 5/29/21 7:09 PM, cal wrote:
> On 5/29/21 5:42 PM, the...@sys-concept.com wrote:
 Another mystery.
 I copied the file to USB 1TB sandisk.  
 md5sum check OK same as my computer 


 md5sum 
 /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
 6f3348f1fb915af9c45806d947558a37  
 /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova

 I mount the same USB 1TB sandisk on another computer and running md5sum on 
 same file gives me different number,  why???

 md5sum 
 /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
 c478cb48e2f7961cb0e3eb452df6e642  
 /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova

>>> Did you sync and unmount the partition before ejecting the drive from
>>> the first computer?  With a file this large being copied, it is likely
>>> that a large amount of data remains buffered/cached and will not be
>>> fully written to the flash memory even after the copy command completes.
>>>
>>> On the first machine, you would still see the correct md5sum because the
>>> kernel abstracts this fact away from you.  But if you rip out the drive
>>> and take it somewhere else without flushing those caches, you're going
>>> to get an incomplete file.
>>>
>>> Check if the file on the drive still md5sums the same if you plug it
>>> back into the first machine.  Check what size it is, and whether there
>>> are a lot of 0s at the end indicating an unfinished write.
>>>
>>> cal
>>
>> Yes, I unmounted the USB device every time. 
>> And yes, I plug the USB device back to original machine and md5sum is 
>> correct, same as the original. 
>>
>> I copied the large file over network to another box and md5sum of:  
>> windows-7_pro_May-23-21.ova is correct same as on the original box.
>>
>> I run this:   "rsync -avh [source] [destination] && rsync -avhc [source] 
>> [destination]"
>>
>> above code rsync files folder on first run and if complete without issue, 
>> will run rsync again immediately while performing same file name comparison 
>> by using hash of entire file.
>>
>> This i what I got:
>>
>> rsync -avh windows-7_pro_May-29-21.ova fd@10.0.0.138:/home/fd/business/VDI/ 
>> && rsync -avhc windows-7_pro_May-29-21.ova 
>> fd@10.0.0.138:/home/fd/business/VDI/
>> sending incremental file list
>> windows-7_pro_May-29-21.ova
>>
>> sent 30.29G bytes  received 35 bytes  115.81M bytes/sec
>> total size is 30.28G  speedup is 1.00
>> sending incremental file list
>> windows-7_pro_May-29-21.ova
>> WARNING: windows-7_pro_May-29-21.ova failed verification -- update discarded 
>> (will try again).
>> windows-7_pro_May-29-21.ova
>> ERROR: windows-7_pro_May-29-21.ova failed verification -- update discarded.
>>
>> sent 33.44M bytes  received 6.47M bytes  123.38K bytes/sec
>> total size is 30.28G  speedup is 758.62
>> rsync error: some files/attrs were not transferred (see previous errors) 
>> (code 23) at main.c(1330) [sender=3.2.3]
>>
>>  
>>
> rsync is emitting errors indicating the file was not transferred
> correctly.  At this point I would call into question whether your second
> machine is the problem rather than any of the tools you're using.  If
> you have a third machine that is easy to test.  Otherwise I would run
> memtest86+ and smartctl.
> 
> cal

Right On Cal, running memtest86+ gave me nothing but errors.  I'm surprised it 
compiled all the packages recently without any errors.
Putting two good stick in it and md5sum worked without a problem.   
Was able to import OVA into virtualbox without any errors. 
 

Re: [gentoo-user] is "scp" reliable?

[cal]

On 5/29/21 5:42 PM, the...@sys-concept.com wrote:
>>> Another mystery.
>>> I copied the file to USB 1TB sandisk.  
>>> md5sum check OK same as my computer 
>>>
>>>
>>> md5sum 
>>> /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
>>> 6f3348f1fb915af9c45806d947558a37  
>>> /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>>
>>> I mount the same USB 1TB sandisk on another computer and running md5sum on 
>>> same file gives me different number,  why???
>>>
>>> md5sum 
>>> /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
>>> c478cb48e2f7961cb0e3eb452df6e642  
>>> /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>>
>> Did you sync and unmount the partition before ejecting the drive from
>> the first computer?  With a file this large being copied, it is likely
>> that a large amount of data remains buffered/cached and will not be
>> fully written to the flash memory even after the copy command completes.
>>
>> On the first machine, you would still see the correct md5sum because the
>> kernel abstracts this fact away from you.  But if you rip out the drive
>> and take it somewhere else without flushing those caches, you're going
>> to get an incomplete file.
>>
>> Check if the file on the drive still md5sums the same if you plug it
>> back into the first machine.  Check what size it is, and whether there
>> are a lot of 0s at the end indicating an unfinished write.
>>
>> cal
> 
> Yes, I unmounted the USB device every time. 
> And yes, I plug the USB device back to original machine and md5sum is 
> correct, same as the original. 
> 
> I copied the large file over network to another box and md5sum of:  
> windows-7_pro_May-23-21.ova is correct same as on the original box.
> 
> I run this:   "rsync -avh [source] [destination] && rsync -avhc [source] 
> [destination]"
> 
> above code rsync files folder on first run and if complete without issue, 
> will run rsync again immediately while performing same file name comparison 
> by using hash of entire file.
> 
> This i what I got:
> 
> rsync -avh windows-7_pro_May-29-21.ova fd@10.0.0.138:/home/fd/business/VDI/ 
> && rsync -avhc windows-7_pro_May-29-21.ova 
> fd@10.0.0.138:/home/fd/business/VDI/
> sending incremental file list
> windows-7_pro_May-29-21.ova
> 
> sent 30.29G bytes  received 35 bytes  115.81M bytes/sec
> total size is 30.28G  speedup is 1.00
> sending incremental file list
> windows-7_pro_May-29-21.ova
> WARNING: windows-7_pro_May-29-21.ova failed verification -- update discarded 
> (will try again).
> windows-7_pro_May-29-21.ova
> ERROR: windows-7_pro_May-29-21.ova failed verification -- update discarded.
> 
> sent 33.44M bytes  received 6.47M bytes  123.38K bytes/sec
> total size is 30.28G  speedup is 758.62
> rsync error: some files/attrs were not transferred (see previous errors) 
> (code 23) at main.c(1330) [sender=3.2.3]
> 
>  
> 
rsync is emitting errors indicating the file was not transferred
correctly.  At this point I would call into question whether your second
machine is the problem rather than any of the tools you're using.  If
you have a third machine that is easy to test.  Otherwise I would run
memtest86+ and smartctl.

cal

Re: [gentoo-user] is "scp" reliable?

[thelma]

>> Another mystery.
>> I copied the file to USB 1TB sandisk.  
>> md5sum check OK same as my computer 
>>
>>
>> md5sum 
>> /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
>> 6f3348f1fb915af9c45806d947558a37  
>> /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>
>> I mount the same USB 1TB sandisk on another computer and running md5sum on 
>> same file gives me different number,  why???
>>
>> md5sum /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
>> c478cb48e2f7961cb0e3eb452df6e642  
>> /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>
> Did you sync and unmount the partition before ejecting the drive from
> the first computer?  With a file this large being copied, it is likely
> that a large amount of data remains buffered/cached and will not be
> fully written to the flash memory even after the copy command completes.
> 
> On the first machine, you would still see the correct md5sum because the
> kernel abstracts this fact away from you.  But if you rip out the drive
> and take it somewhere else without flushing those caches, you're going
> to get an incomplete file.
> 
> Check if the file on the drive still md5sums the same if you plug it
> back into the first machine.  Check what size it is, and whether there
> are a lot of 0s at the end indicating an unfinished write.
> 
> cal

Yes, I unmounted the USB device every time. 
And yes, I plug the USB device back to original machine and md5sum is correct, 
same as the original. 

I copied the large file over network to another box and md5sum of:  
windows-7_pro_May-23-21.ova is correct same as on the original box.

I run this:   "rsync -avh [source] [destination] && rsync -avhc [source] 
[destination]"

above code rsync files folder on first run and if complete without issue, will 
run rsync again immediately while performing same file name comparison by using 
hash of entire file.

This i what I got:

rsync -avh windows-7_pro_May-29-21.ova fd@10.0.0.138:/home/fd/business/VDI/ && 
rsync -avhc windows-7_pro_May-29-21.ova fd@10.0.0.138:/home/fd/business/VDI/
sending incremental file list
windows-7_pro_May-29-21.ova

sent 30.29G bytes  received 35 bytes  115.81M bytes/sec
total size is 30.28G  speedup is 1.00
sending incremental file list
windows-7_pro_May-29-21.ova
WARNING: windows-7_pro_May-29-21.ova failed verification -- update discarded 
(will try again).
windows-7_pro_May-29-21.ova
ERROR: windows-7_pro_May-29-21.ova failed verification -- update discarded.

sent 33.44M bytes  received 6.47M bytes  123.38K bytes/sec
total size is 30.28G  speedup is 758.62
rsync error: some files/attrs were not transferred (see previous errors) (code 
23) at main.c(1330) [sender=3.2.3]

 

Re: [gentoo-user] is "scp" reliable?

[thelma]

On 5/29/21 3:32 PM, Mark Knecht wrote:
> 
> 
> On Sat, May 29, 2021 at 2:04 PM Mark Knecht  > wrote:
>>
>>
>>
>> On Sat, May 29, 2021 at 1:33 PM > > wrote:
>> 
>> > Another mystery.
>> > I copied the file to USB 1TB sandisk.
>> > md5sum check OK same as my computer
>> >
>> >
>> > md5sum 
>> > /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>> > 6f3348f1fb915af9c45806d947558a37  
>> > /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>> >
>> > I mount the same USB 1TB sandisk on another computer and running md5sum on 
>> > same file gives me different number,  why???
>> >
>> > md5sum 
>> > /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>> > c478cb48e2f7961cb0e3eb452df6e642  
>> > /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>
>> Different revisions of md5sum possibly?
>>
>> - Mark
> 
> 
> On the first machine did the md5sum of the file on the internal hard drive 
> match the version on the USB device? 

Yes, the md5sum output on the  internal HD is the same as on the USB device.
 

Re: [gentoo-user] is "scp" reliable?

[cal]

On 5/29/21 1:32 PM, the...@sys-concept.com wrote:
> On 5/29/21 2:00 PM, Mark Knecht wrote:
>>
>>
>> On Sat, May 29, 2021 at 12:54 PM > > wrote:
>>>
>>> I use "scp" to copy large file over local network, windows-7_pro.ova (about 
>>> 28GB) but I it failed to import the appliance with VirtualBox.
>>> Checking the md5sum of the source and destination it turn out they are 
>>> different.
>>>
>>> Is "scp" reliable for large files?
>>
>> In my experience scp is reliable but for large files I'd probably opt for 
>> some version of rsync, possibly the -P option. That way if the transfer 
>> stops for some reason you can generally pick up where it left off.
>>
>> HTH,
>> Mark
> 
> Another mystery.
> I copied the file to USB 1TB sandisk.  
> md5sum check OK same as my computer 
> 
> 
> md5sum 
> /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
> 6f3348f1fb915af9c45806d947558a37  
> /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> 
> I mount the same USB 1TB sandisk on another computer and running md5sum on 
> same file gives me different number,  why???
> 
> md5sum /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
> c478cb48e2f7961cb0e3eb452df6e642  
> /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> 
Did you sync and unmount the partition before ejecting the drive from
the first computer?  With a file this large being copied, it is likely
that a large amount of data remains buffered/cached and will not be
fully written to the flash memory even after the copy command completes.

On the first machine, you would still see the correct md5sum because the
kernel abstracts this fact away from you.  But if you rip out the drive
and take it somewhere else without flushing those caches, you're going
to get an incomplete file.

Check if the file on the drive still md5sums the same if you plug it
back into the first machine.  Check what size it is, and whether there
are a lot of 0s at the end indicating an unfinished write.

cal

Re: [gentoo-user] is "scp" reliable?

[Mark Knecht]

On Sat, May 29, 2021 at 2:04 PM Mark Knecht  wrote:
>
>
>
> On Sat, May 29, 2021 at 1:33 PM  wrote:
> 
> > Another mystery.
> > I copied the file to USB 1TB sandisk.
> > md5sum check OK same as my computer
> >
> >
> > md5sum
/run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> > 6f3348f1fb915af9c45806d947558a37
 /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> >
> > I mount the same USB 1TB sandisk on another computer and running md5sum
on same file gives me different number,  why???
> >
> > md5sum
/run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> > c478cb48e2f7961cb0e3eb452df6e642
 /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>
> Different revisions of md5sum possibly?
>
> - Mark

>
On the first machine did the md5sum of the file on the internal hard drive
match the version on the USB device?

I don't know if the underlying file system type effects this calculation.
I'd hope not but I don't know.

Re: [gentoo-user] is "scp" reliable?

[Mark Knecht]

On Sat, May 29, 2021 at 1:33 PM  wrote:

> Another mystery.
> I copied the file to USB 1TB sandisk.
> md5sum check OK same as my computer
>
>
> md5sum
/run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> 6f3348f1fb915af9c45806d947558a37
 /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>
> I mount the same USB 1TB sandisk on another computer and running md5sum
on same file gives me different number,  why???
>
> md5sum
/run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> c478cb48e2f7961cb0e3eb452df6e642
 /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova

Different revisions of md5sum possibly?

- Mark

Re: [gentoo-user] is "scp" reliable?

[thelma]

On 5/29/21 2:00 PM, Mark Knecht wrote:
> 
> 
> On Sat, May 29, 2021 at 12:54 PM  > wrote:
>>
>> I use "scp" to copy large file over local network, windows-7_pro.ova (about 
>> 28GB) but I it failed to import the appliance with VirtualBox.
>> Checking the md5sum of the source and destination it turn out they are 
>> different.
>>
>> Is "scp" reliable for large files?
> 
> In my experience scp is reliable but for large files I'd probably opt for 
> some version of rsync, possibly the -P option. That way if the transfer stops 
> for some reason you can generally pick up where it left off.
> 
> HTH,
> Mark

Another mystery.
I copied the file to USB 1TB sandisk.  
md5sum check OK same as my computer 


md5sum 
/run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
6f3348f1fb915af9c45806d947558a37  
/run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova

I mount the same USB 1TB sandisk on another computer and running md5sum on same 
file gives me different number,  why???

md5sum /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
c478cb48e2f7961cb0e3eb452df6e642  
/run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova

Re: [gentoo-user] is "scp" reliable?

[Mark Knecht]

On Sat, May 29, 2021 at 12:54 PM  wrote:
>
> I use "scp" to copy large file over local network, windows-7_pro.ova
(about 28GB) but I it failed to import the appliance with VirtualBox.
> Checking the md5sum of the source and destination it turn out they are
different.
>
> Is "scp" reliable for large files?

In my experience scp is reliable but for large files I'd probably opt for
some version of rsync, possibly the -P option. That way if the transfer
stops for some reason you can generally pick up where it left off.

HTH,
Mark

Re: [gentoo-user] is "scp" reliable?

[thelma]

On 5/29/21 1:53 PM, the...@sys-concept.com wrote:
> I use "scp" to copy large file over local network, windows-7_pro.ova (about 
> 28GB) but I it failed to import the appliance with VirtualBox.
> Checking the md5sum of the source and destination it turn out they are 
> different.
> 
> Is "scp" reliable for large files? 

Even with "rsync" I get a different result.

Source: md5sum windows-7_pro.ova
55073f564101274bc7cd609a25c21e5b

destination:  md5sum windows-7_pro.ova
f5f3404cf1b0f654e873abe7e21e4328

[gentoo-user] is "scp" reliable?

[thelma]

I use "scp" to copy large file over local network, windows-7_pro.ova (about 
28GB) but I it failed to import the appliance with VirtualBox.
Checking the md5sum of the source and destination it turn out they are 
different.

Is "scp" reliable for large files?