Re: signify and ftp.eu.openbsd.org

[Janne Johansson]

2018-04-11 22:44 GMT+02:00 Peter J. Philipp :

> On Wed, Apr 11, 2018 at 10:20:37PM +0200, Paul de Weerd wrote:
> > ftp doesn't do this itself, but the error detection in tcp and ssl
> > (ok, so that's linked into the ftp binary) do.
> >
> > The file is unlikely to have been changed in flight.
>
> OK, odd. What I find odd is that the SHA256.sig file I got (which was
> downloaded after the .iso) differs from the very first time I made contact
> with ftp.eu
> today with the SHA256.sig that I downloaded to verify.  Could this be an
> atomicity thing with the FTP mirror?
>

The mirror tries to download and replace files dir-by-dir, but not edit
files in-place, so its not impossible to hit the window
when one of the files but not the other is replaced even if its normally a
very small chance. Each snapshot is something
like 100G, so there just is no way (for me) to atomically replace the lot
so everything in between is a tradeoff between
disk-usage, no instant writeable snaps and simplicity.

-- 
May the most significant bit of your life be positive.

Re: signify and ftp.eu.openbsd.org

[Paul de Weerd]

On Wed, Apr 11, 2018 at 10:44:28PM +0200, Peter J. Philipp wrote:
| On Wed, Apr 11, 2018 at 10:20:37PM +0200, Paul de Weerd wrote:
| > ftp doesn't do this itself, but the error detection in tcp and ssl
| > (ok, so that's linked into the ftp binary) do.
| > 
| > The file is unlikely to have been changed in flight.
| 
| OK, odd. What I find odd is that the SHA256.sig file I got (which
| was downloaded after the .iso) differs from the very first time I
| made contact with ftp.eu today with the SHA256.sig that I downloaded
| to verify.  Could this be an atomicity thing with the FTP mirror?

So you start your download of the ISO while the mirror is in the
process of updating from upstream.  The download takes some time,
maybe five minutes (which would be ~1MB/s).  In the mean time the
mirror finishes its update to the latest snap (probably using rsync's
--delay-updates flag), but it keeps sending you the file that was just
deleted (your download is "atomic").  Then you download the (new)
SHA256.sig.

Then the ISO will be from an older snap and it will fail verification.
Snapshots are produced so often (since I started keeping a backlog of
old kernels back in 2011, I've seen 18 days with 4 amd64 snaps *from
the same day*, 170 days with 3 amd64 snaps... I update 4 times a day)
that this is actually not an unlikely scenario.

I think this is what happened to you.  The only way I can think of to
guard against this is by downloading the .sig file twice, both before
and after downloading other files.  If they're different, you hit this
corner case.

Sorry, no big conspiracy where 3-letter agencies are trying to
sabotage the installs of our favorite OS ;-)  Your checking of the
signify signature and the SHA256 checksum does help to detect such
sabotage, but it's still not very likely.

Cheers,

Paul 'WEiRD' de Weerd

-- 
>[<++>-]<+++.>+++[<-->-]<.>+++[<+
+++>-]<.>++[<>-]<+.--.[-]
 http://www.weirdnet.nl/ 

Re: signify and ftp.eu.openbsd.org

[Peter J. Philipp]

On Wed, Apr 11, 2018 at 10:20:37PM +0200, Paul de Weerd wrote:
> ftp doesn't do this itself, but the error detection in tcp and ssl
> (ok, so that's linked into the ftp binary) do.
> 
> The file is unlikely to have been changed in flight.

OK, odd. What I find odd is that the SHA256.sig file I got (which was 
downloaded after the .iso) differs from the very first time I made contact with 
ftp.eu
today with the SHA256.sig that I downloaded to verify.  Could this be an
atomicity thing with the FTP mirror?



> | 4. Is it possible that there was an attack on the OpenBSD network 
> infrastructure?
> 
> Unlikely.  Probably, the bytes you got match 1:1 with the file I got,
> minus those last 4096 bytes.  There's probably nothing all that
> interesting in those last 4K of the ISO, so my guess would be that
> everything would still be fine.  You can verify that by checking the
> contents of the mounted file system against the SHA256 file.
> 
> | 5. Or mine?
> 
> Also unlikely.  Could the download have been interrupted at the last
> moment?

Not by me with any signals, I let the download finish.  Theo does want me to
investigate this box because it has problems waking/sleeping, and I was going
to do that tonight but now it's too late, this signature anomaly wasted my
time for tonight.  It's entirely possible the drive is dying on it.  After
swapping SATA cables I think I'll put an SSD in it.

Thanks Paul (and everyone else in this thread).

-peter

> Cheers,
> 
> Paul

Re: signify and ftp.eu.openbsd.org

[Paul de Weerd]

On Wed, Apr 11, 2018 at 09:52:41PM +0200, Peter J. Philipp wrote:
| So I re-downloaded this file on another machine.  It turned out to have your
| checksum (sha256 -b) and upon downloading another SHA256.sig it signify'ed 
| correctly.  But this leads me to ask questions:
| 
| 1. doesn't the https mode ftp use some sort of authentication/integrity 
| checking while it's downloading?  It's encrypted and anyone with a bit of
| knowledge knows that if encryption is changed in-flight it will flip bits,
| but with a MAC/HMAC/GMAC it should detect any anomalies.  Does ftp not check
| for this?

ftp doesn't do this itself, but the error detection in tcp and ssl
(ok, so that's linked into the ftp binary) do.

The file is unlikely to have been changed in flight.

| 2. I use quite recently a new unencrypted wifi network but I have IPSEC 
enabled
| which uses aes-256 and auth hmac-sha2-256:
| 
| SAD:
| esp tunnel from fd43:5602:29bd:16:0:dead:beef:16 to 
fd43:5602:29bd:16:0:dead:beef:1 spi 0x6ab04db8 auth hmac-sha2-256 enc aes-256
| esp tunnel from fd43:5602:29bd:16:0:dead:beef:1 to 
fd43:5602:29bd:16:0:dead:beef:16 spi 0x780a14d0 auth hmac-sha2-256 enc aes-256
| 
| It is entirely possible that my IPSEC was off minutely and the gif tunnel 
inside it was not protected, but there was still the https...

This is probably also not relevant to the result you got.

| 3. The argument that this may have been modified by ftpmirror (or similar 
software) is good but I stat'ed the file and it's larger than what was 
downloaded by
| us on the second try.  A cmp -l shows a lot of differences... here is the 
stat:
| theta$ touch test
| theta$ date
| Wed Apr 11 21:45:16 CEST 2018
| theta$ stat test
| 1034 8616981 -rw-r--r-- 1 pjp pjp 0 0 "Apr 11 21:45:15 2018" "Apr 11 21:45:15 
2018" "Apr 11 21:45:15 2018" 32768 0 0 test
| theta$ stat install63.iso
| 1034 8616961 -rw-r--r-- 1 pjp pjp 34424768 355905536 "Apr 11 20:59:23 2018" 
"Apr 11 19:33:47 2018" "Apr 11 19:33:47 2018" 32768 695360 0 install63.iso

That's actually *smaller* than what I have here.  The install63.iso I
got is 355909632 bytes in size.  Your file is missing 4096 bytes.

| I did mount this iso with vnconfig is it possible that it was modified then?  
| (and grew?).

No, vnd's don't grow in size during mount.  Their atime may change,
but I don't think much else will (not in the case of ISOs, which are
read-only by nature).

| 4. Is it possible that there was an attack on the OpenBSD network 
infrastructure?

Unlikely.  Probably, the bytes you got match 1:1 with the file I got,
minus those last 4096 bytes.  There's probably nothing all that
interesting in those last 4K of the ISO, so my guess would be that
everything would still be fine.  You can verify that by checking the
contents of the mounted file system against the SHA256 file.

| 5. Or mine?

Also unlikely.  Could the download have been interrupted at the last
moment?

Cheers,

Paul

-- 
>[<++>-]<+++.>+++[<-->-]<.>+++[<+
+++>-]<.>++[<>-]<+.--.[-]
 http://www.weirdnet.nl/ 

Re: signify and ftp.eu.openbsd.org

[Peter J. Philipp]

On Wed, Apr 11, 2018 at 08:45:40PM +0200, Paul de Weerd wrote:
> Hi Peter,

Hello Paul,

> I downloaded those exact two files from the same IP addresses and the
> signature verified OK for me:
> 
> [weerd@pom] $ ftp -4 https://ftp.eu.openbsd.org/pub/OpenBSD/snapshots/amd64/i 
> >
> Trying 193.156.26.18...
> Requesting 
> https://ftp.eu.openbsd.org/pub/OpenBSD/snapshots/amd64/install63.iso
> 100% |**|   339 MB 01:41
> 355909632 bytes received in 101.67 seconds (3.34 MB/s)
> [weerd@pom] $ nBSD/snapshots/amd64/SHA256.sig 
> <
> Trying 193.156.26.18...
> Requesting
> https://ftp.eu.openbsd.org/pub/OpenBSD/snapshots/amd64/SHA256.sig
> 100% |**|  2152 00:00
> 2152 bytes received in 0.00 seconds (1.02 MB/s)
> [weerd@pom] $ sd-63-base.pub -x SHA256.sig install63.iso  
> <
> Signature Verified
> install63.iso: OK
> 
> (Note that I forced IPv4 as IPv6 was rather slow for me, but both
> addresses match what your dig gave you).
> 
> So, some options:
> 
> Maybe you caught the mirror mid-update.  Perhaps a bit fell over due
> to cosmic radiation hitting your machine at the wrong moment,
> affecting a bit of RAM.  Maybe your storage medium is dying.
> 
> Could be various reasons why it failed; can you try again and see if
> it still fails?  Try also on another machine, if you have one around.
> I got WxAW3clMg3BLs/NBq58q9lMGlWFQLAOW5ToeltQlSyU= as a sha256 hash.
> 
> Cheers,
> 
> Paul 'WEiRD' de Weerd

So I re-downloaded this file on another machine.  It turned out to have your
checksum (sha256 -b) and upon downloading another SHA256.sig it signify'ed 
correctly.  But this leads me to ask questions:

1. doesn't the https mode ftp use some sort of authentication/integrity 
checking while it's downloading?  It's encrypted and anyone with a bit of
knowledge knows that if encryption is changed in-flight it will flip bits,
but with a MAC/HMAC/GMAC it should detect any anomalies.  Does ftp not check
for this?

2. I use quite recently a new unencrypted wifi network but I have IPSEC enabled
which uses aes-256 and auth hmac-sha2-256:

SAD:
esp tunnel from fd43:5602:29bd:16:0:dead:beef:16 to 
fd43:5602:29bd:16:0:dead:beef:1 spi 0x6ab04db8 auth hmac-sha2-256 enc aes-256
esp tunnel from fd43:5602:29bd:16:0:dead:beef:1 to 
fd43:5602:29bd:16:0:dead:beef:16 spi 0x780a14d0 auth hmac-sha2-256 enc aes-256

It is entirely possible that my IPSEC was off minutely and the gif tunnel 
inside it was not protected, but there was still the https...

3. The argument that this may have been modified by ftpmirror (or similar 
software) is good but I stat'ed the file and it's larger than what was 
downloaded by
us on the second try.  A cmp -l shows a lot of differences... here is the stat:
theta$ touch test
theta$ date
Wed Apr 11 21:45:16 CEST 2018
theta$ stat test
1034 8616981 -rw-r--r-- 1 pjp pjp 0 0 "Apr 11 21:45:15 2018" "Apr 11 21:45:15 
2018" "Apr 11 21:45:15 2018" 32768 0 0 test
theta$ stat install63.iso
1034 8616961 -rw-r--r-- 1 pjp pjp 34424768 355905536 "Apr 11 20:59:23 2018" 
"Apr 11 19:33:47 2018" "Apr 11 19:33:47 2018" 32768 695360 0 install63.iso

I did mount this iso with vnconfig is it possible that it was modified then?  
(and grew?).

4. Is it possible that there was an attack on the OpenBSD network 
infrastructure?

5. Or mine?

Best regards,
-peter

Re: signify and ftp.eu.openbsd.org

[Stuart Henderson]

On 2018-04-11, Peter J. Philipp <p...@centroid.eu> wrote:
> Sorry to be writing this twice.  Not my day.
>
> The complete path where I downloaded this from was
> https://ftp.eu.openbsd.org/pub/OpenBSD/snapshots/amd64/install63.iso
>
> including the SHA256.sig which I check against.
>
> Also:
>
> beta$ dig ftp.eu.openbsd.org +short
> 193.156.26.18
> beta$ dig ftp.eu.openbsd.org +short 
> 2001:700:3:4017::100
>
> that is in my DNS cache is this all correct?
>
> Best Regards,
>
> -peter
>
>
> On 04/11/18 19:54, Peter J. Philipp wrote:
>> Hi,
>>
>> I just downloaded this install63.iso from https://ftp.eu.openbsd.org:
>>
>> beta$  signify -C -p /etc/signify/openbsd-63-base.pub \   
>>> -x SHA256.sig bsd
>> Signature Verified
>> bsd: OK
>> beta$  signify -C -p /etc/signify/openbsd-63-base.pub \ 
>>> -x SHA256.sig install63.iso
>> Signature Verified
>> install63.iso: FAIL
>> beta$ sha256 -C SHA256 install63.iso 
>>   
>> (SHA256) install63.iso: FAILED
>>
>> What's going on ?  Why has the checksum failed?
>>
>> Regards,
>> -peter
>
>

A new snapshot may have been produced while the files were transferring,
so SHA256.sig is from one snapshot and install63.iso from another.

If you try another mirror at the same time and get the same error and
file contents from both, it's quite likely that is what happened.

Re: signify and ftp.eu.openbsd.org

[Paul de Weerd]

Hi Peter,

On Wed, Apr 11, 2018 at 07:58:56PM +0200, Peter J. Philipp wrote:
| Sorry to be writing this twice.  Not my day.
| 
| The complete path where I downloaded this from was
| https://ftp.eu.openbsd.org/pub/OpenBSD/snapshots/amd64/install63.iso
| 
| including the SHA256.sig which I check against.
| 
| Also:
| 
| beta$ dig ftp.eu.openbsd.org +short
| 193.156.26.18
| beta$ dig ftp.eu.openbsd.org +short 
| 2001:700:3:4017::100

I downloaded those exact two files from the same IP addresses and the
signature verified OK for me:

[weerd@pom] $ ftp -4 https://ftp.eu.openbsd.org/pub/OpenBSD/snapshots/amd64/i >
Trying 193.156.26.18...
Requesting https://ftp.eu.openbsd.org/pub/OpenBSD/snapshots/amd64/install63.iso
100% |**|   339 MB 01:41
355909632 bytes received in 101.67 seconds (3.34 MB/s)
[weerd@pom] $ nBSD/snapshots/amd64/SHA256.sig <
Trying 193.156.26.18...
Requesting
https://ftp.eu.openbsd.org/pub/OpenBSD/snapshots/amd64/SHA256.sig
100% |**|  2152 00:00
2152 bytes received in 0.00 seconds (1.02 MB/s)
[weerd@pom] $ sd-63-base.pub -x SHA256.sig install63.iso  <
Signature Verified
install63.iso: OK

(Note that I forced IPv4 as IPv6 was rather slow for me, but both
addresses match what your dig gave you).

So, some options:

Maybe you caught the mirror mid-update.  Perhaps a bit fell over due
to cosmic radiation hitting your machine at the wrong moment,
affecting a bit of RAM.  Maybe your storage medium is dying.

Could be various reasons why it failed; can you try again and see if
it still fails?  Try also on another machine, if you have one around.
I got WxAW3clMg3BLs/NBq58q9lMGlWFQLAOW5ToeltQlSyU= as a sha256 hash.

Cheers,

Paul 'WEiRD' de Weerd

| that is in my DNS cache is this all correct?
| 
| Best Regards,
| 
| -peter
| 
| 
| On 04/11/18 19:54, Peter J. Philipp wrote:
| > Hi,
| >
| > I just downloaded this install63.iso from https://ftp.eu.openbsd.org:
| >
| > beta$  signify -C -p /etc/signify/openbsd-63-base.pub \   
| >> -x SHA256.sig bsd
| > Signature Verified
| > bsd: OK
| > beta$  signify -C -p /etc/signify/openbsd-63-base.pub \ 
| >> -x SHA256.sig install63.iso
| > Signature Verified
| > install63.iso: FAIL
| > beta$ sha256 -C SHA256 install63.iso
   
| > (SHA256) install63.iso: FAILED
| >
| > What's going on ?  Why has the checksum failed?
| >
| > Regards,
| > -peter
| 

-- 
>[<++>-]<+++.>+++[<-->-]<.>+++[<+
+++>-]<.>++[<>-]<+.--.[-]
 http://www.weirdnet.nl/ 

Re: signify and ftp.eu.openbsd.org

[Peter J. Philipp]

Sorry to be writing this twice.  Not my day.

The complete path where I downloaded this from was
https://ftp.eu.openbsd.org/pub/OpenBSD/snapshots/amd64/install63.iso

including the SHA256.sig which I check against.

Also:

beta$ dig ftp.eu.openbsd.org +short
193.156.26.18
beta$ dig ftp.eu.openbsd.org +short 
2001:700:3:4017::100

that is in my DNS cache is this all correct?

Best Regards,

-peter


On 04/11/18 19:54, Peter J. Philipp wrote:
> Hi,
>
> I just downloaded this install63.iso from https://ftp.eu.openbsd.org:
>
> beta$  signify -C -p /etc/signify/openbsd-63-base.pub \   
>> -x SHA256.sig bsd
> Signature Verified
> bsd: OK
> beta$  signify -C -p /etc/signify/openbsd-63-base.pub \ 
>> -x SHA256.sig install63.iso
> Signature Verified
> install63.iso: FAIL
> beta$ sha256 -C SHA256 install63.iso  
>  
> (SHA256) install63.iso: FAILED
>
> What's going on ?  Why has the checksum failed?
>
> Regards,
> -peter

signify and ftp.eu.openbsd.org

[Peter J. Philipp]

Hi,

I just downloaded this install63.iso from https://ftp.eu.openbsd.org:

beta$  signify -C -p /etc/signify/openbsd-63-base.pub \   
> -x SHA256.sig bsd
Signature Verified
bsd: OK
beta$  signify -C -p /etc/signify/openbsd-63-base.pub \ 
> -x SHA256.sig install63.iso
Signature Verified
install63.iso: FAIL
beta$ sha256 -C SHA256 install63.iso   
(SHA256) install63.iso: FAILED

What's going on ?  Why has the checksum failed?

Regards,
-peter