Bug#840104: Encrypted uploads to the security archive

[Aurelien Jarno]

On 2018-02-13 07:22, Aurelien Jarno wrote:
> On 2018-02-01 22:17, Ansgar Burchardt wrote:
> > Philipp Kern writes:
> > > On 01.02.2018 10:30, Ansgar Burchardt wrote:
> > [...]
> > >> There is already a `buildd-uploader` role account on the upload hosts
> > >> both main and security archive, a `rsync-ssh-wrap` script, and someone
> > >> also set up authorized_keys.
> > >> 
> > >> I'm just not sure if it is already in use for security uploads?  I
> > >> believe it was used for uploads to the main archive already (not sure if
> > >> it currently is?).
> > >
> > > Indeed, it uses rsync over SSH through dupload. For security it uses
> > > FTP. Interestingly an rsync-security dupload.conf entry exists, but it
> > > doesn't seem to be used[1].
> > 
> > Hmm, maybe we should try if it does the right thing?  The wrapper script
> > should ignore the `chmod` call I mentioned in #876900, so the uploaded
> > files shouldn't even be readable by other DDs.

Note that the chmod has been ignored in the wrapper script since almost
the beginning of its existence.

> The problem there is that rsync when used with dupload forces the
> uploaded file to be world readable, until the package is moved out from
> the upload directory by dupload.

I have found a way to force rsync permissions to 0640. I have applied
that to the wrapper script. Following that I have switched the upload
queue on the build daemons to the SSH one.

I guess this basically solves this bug.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net

Bug#840104: Encrypted uploads to the security archive

[Aurelien Jarno]

On 2018-02-01 20:45, Philipp Kern wrote:
> On 01.02.2018 10:30, Ansgar Burchardt wrote:
> > Hmm, another issue comes to mind:
> > 
> > If we care about encrypted buildd uploads, the buildds should probably
> > also download from the (private) security-buildd archive over an
> > encrypted connection, ideally with client authentication.  Otherwise
> > people could see the embargoed fixes in the source package.
> 
> Well, I thought this was already the case at this point. I suppose it
> shouldn't be too hard to add https:// support at this point given that
> apt supports it natively. But I think client auth should be a weak
> requirement at this point.

Since a few hours ago the build daemons access the security archive in
https. This might not be the perfect solution, but it's already an
improvement compared to plain http:// and it was (relatively) easy to
do. It doesn't prevent looking for a better solution though.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net

Bug#840104: Encrypted uploads to the security archive

[Aurelien Jarno]

On 2018-02-01 22:17, Ansgar Burchardt wrote:
> Philipp Kern writes:
> > On 01.02.2018 10:30, Ansgar Burchardt wrote:
> [...]
> >> There is already a `buildd-uploader` role account on the upload hosts
> >> both main and security archive, a `rsync-ssh-wrap` script, and someone
> >> also set up authorized_keys.
> >> 
> >> I'm just not sure if it is already in use for security uploads?  I
> >> believe it was used for uploads to the main archive already (not sure if
> >> it currently is?).
> >
> > Indeed, it uses rsync over SSH through dupload. For security it uses
> > FTP. Interestingly an rsync-security dupload.conf entry exists, but it
> > doesn't seem to be used[1].
> 
> Hmm, maybe we should try if it does the right thing?  The wrapper script
> should ignore the `chmod` call I mentioned in #876900, so the uploaded
> files shouldn't even be readable by other DDs.

The problem there is that rsync when used with dupload forces the
uploaded file to be world readable, until the package is moved out from
the upload directory by dupload.

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net

Bug#840104: Encrypted uploads to the security archive

[Ansgar Burchardt]

Philipp Kern writes:
> On 01.02.2018 10:30, Ansgar Burchardt wrote:
[...]
>> There is already a `buildd-uploader` role account on the upload hosts
>> both main and security archive, a `rsync-ssh-wrap` script, and someone
>> also set up authorized_keys.
>> 
>> I'm just not sure if it is already in use for security uploads?  I
>> believe it was used for uploads to the main archive already (not sure if
>> it currently is?).
>
> Indeed, it uses rsync over SSH through dupload. For security it uses
> FTP. Interestingly an rsync-security dupload.conf entry exists, but it
> doesn't seem to be used[1].

Hmm, maybe we should try if it does the right thing?  The wrapper script
should ignore the `chmod` call I mentioned in #876900, so the uploaded
files shouldn't even be readable by other DDs.

Please watch file permissions of the files on the upload host during
upload and after upload (before debianqueued processes them) if you try.

>> Hmm, another issue comes to mind:
>> 
>> If we care about encrypted buildd uploads, the buildds should probably
>> also download from the (private) security-buildd archive over an
>> encrypted connection, ideally with client authentication.  Otherwise
>> people could see the embargoed fixes in the source package.
>
> Well, I thought this was already the case at this point. I suppose it
> shouldn't be too hard to add https:// support at this point given that
> apt supports it natively. But I think client auth should be a weak
> requirement at this point.

If someone can read from the buildd <-> security-master connection, he
probably can also sent packages with the buildd's IP.  So just https://
doesn't bring much IMHO.  (It doesn't hurt either.)

>> Maybe a local proxy that translates http to https requests and which
>> provides a client certificate (so the chroot doesn't need it added)?
>> The proxy could also filter out other network requests we don't want by
>> default.
>
> It's the kind of solution that I'd propose as well, if the preconditions
> were different ones. Unless it's a pretty stock setup that can be run
> with default packages I don't see that happening.

I think squid can do that:

  tls_outgoing_options cert= key=: for client certificate
  url_rewrite_*: for http:// -> https:// rewrite

`url_rewrite_program` would need a simple script doing the rewriting.

Of course that still leaves managing the client certificates themselves.

For limiting network access (for programs respecting *_proxy environment
variables), one would need to know which URLs are okay.  That should
pretty much be the list of mirrors any chroot might end up using.  This
needs to be generated automatically as buildds use different mirrors.

I use squid for limiting network access for GitLab CI runners which
works quite nice.  I haven't used client certificates or URL rewriting
with squid so far.  Maybe I should try if it works as expected.

Ansgar

Bug#840104: Encrypted uploads to the security archive

[Philipp Kern]

On 01.02.2018 10:30, Ansgar Burchardt wrote:
> Philipp Kern writes:
>> On 31.01.2018 01:11, Ansgar Burchardt wrote:
>>> I'm not sure if buildds are already configured to upload to the security
>>> archive via ssh as they do for the main archive.  It might be a good
>>> idea to do so.
>>
>> What's the requirement here? I think traditionally we use machine-local
>> SSH authorized_keys for role accounts. So we already provision keys to
>> every buildd that allows it to talk to wanna-build, but I'm not sure how
>> we'd maintain that with another host. Especially one that presumably can
>> be repointed?
> 
> There is already a `buildd-uploader` role account on the upload hosts
> both main and security archive, a `rsync-ssh-wrap` script, and someone
> also set up authorized_keys.
> 
> I'm just not sure if it is already in use for security uploads?  I
> believe it was used for uploads to the main archive already (not sure if
> it currently is?).

Indeed, it uses rsync over SSH through dupload. For security it uses
FTP. Interestingly an rsync-security dupload.conf entry exists, but it
doesn't seem to be used[1].

>> Maybe this is more of a question for DSA, but I don't know what the
>> current setup entails and if you wrote your own SSH daemon for uploads.
>> In that case we should be able to figure something out.
> 
> It's the regular OpenSSH with forced command setup.
> 
> Hmm, another issue comes to mind:
> 
> If we care about encrypted buildd uploads, the buildds should probably
> also download from the (private) security-buildd archive over an
> encrypted connection, ideally with client authentication.  Otherwise
> people could see the embargoed fixes in the source package.

Well, I thought this was already the case at this point. I suppose it
shouldn't be too hard to add https:// support at this point given that
apt supports it natively. But I think client auth should be a weak
requirement at this point.

> Maybe a local proxy that translates http to https requests and which
> provides a client certificate (so the chroot doesn't need it added)?
> The proxy could also filter out other network requests we don't want by
> default.

It's the kind of solution that I'd propose as well, if the preconditions
were different ones. Unless it's a pretty stock setup that can be run
with default packages I don't see that happening.

> We could also move the security buildd archive away from security-master
> at the same time (as we did for ftp-master).  Then there should no
> longer be a reason for a httpd on security-master.

Whatever works for you, the script is now in puppet[2]. (Assuming that
you'd be able to forward-port the current ACLing system.)

Kind regards and thanks
Philipp Kern

[1]
https://salsa.debian.org/dsa-team/mirror/dsa-puppet/blob/master/modules/buildd/files/dupload.conf
[2]
https://salsa.debian.org/dsa-team/mirror/dsa-puppet/blob/master/modules/schroot/files/schroot-setup.d/99builddsourceslist

Bug#840104: Encrypted uploads to the security archive

[Ansgar Burchardt]

Philipp Kern writes:
> On 31.01.2018 01:11, Ansgar Burchardt wrote:
>> I'm not sure if buildds are already configured to upload to the security
>> archive via ssh as they do for the main archive.  It might be a good
>> idea to do so.
>
> What's the requirement here? I think traditionally we use machine-local
> SSH authorized_keys for role accounts. So we already provision keys to
> every buildd that allows it to talk to wanna-build, but I'm not sure how
> we'd maintain that with another host. Especially one that presumably can
> be repointed?

There is already a `buildd-uploader` role account on the upload hosts
both main and security archive, a `rsync-ssh-wrap` script, and someone
also set up authorized_keys.

I'm just not sure if it is already in use for security uploads?  I
believe it was used for uploads to the main archive already (not sure if
it currently is?).

> Maybe this is more of a question for DSA, but I don't know what the
> current setup entails and if you wrote your own SSH daemon for uploads.
> In that case we should be able to figure something out.

It's the regular OpenSSH with forced command setup.

Hmm, another issue comes to mind:

If we care about encrypted buildd uploads, the buildds should probably
also download from the (private) security-buildd archive over an
encrypted connection, ideally with client authentication.  Otherwise
people could see the embargoed fixes in the source package.

Maybe a local proxy that translates http to https requests and which
provides a client certificate (so the chroot doesn't need it added)?
The proxy could also filter out other network requests we don't want by
default.

We could also move the security buildd archive away from security-master
at the same time (as we did for ftp-master).  Then there should no
longer be a reason for a httpd on security-master.

Ansgar

Bug#840104: Encrypted uploads to the security archive

[Philipp Kern]

On 31.01.2018 01:11, Ansgar Burchardt wrote:
> I'm not sure if buildds are already configured to upload to the security
> archive via ssh as they do for the main archive.  It might be a good
> idea to do so.

What's the requirement here? I think traditionally we use machine-local
SSH authorized_keys for role accounts. So we already provision keys to
every buildd that allows it to talk to wanna-build, but I'm not sure how
we'd maintain that with another host. Especially one that presumably can
be repointed?

Maybe this is more of a question for DSA, but I don't know what the
current setup entails and if you wrote your own SSH daemon for uploads.
In that case we should be able to figure something out.

Alternatively I suppose DSA could also provide something through
stunnel, but then I think we'd be back to encrypted FTP.

Kind regards and thanks
Philipp Kern

Bug#840104: Encrypted uploads to the security archive

[Ansgar Burchardt]

Kurt Roeckx writes:
> It's been a longstanding problem that the uploads to the security
> archive are not encrypted in any way. I think this is a problem
> for all embargoed uploads that we are doing.
>
> Upstream might actually do all that's possible to keep the
> security issues secret. But it can potentionally leak when it gets
> uploaded to the security archive. As far as I know only ftp is
> currently supported.
>
> I can think of several ways of doing this, but you probably want
> to talk to DSA about some of those options. They include:
> - Allow uploads over ssh / sftp. This could be anonymous, or give
>   access to the same user with all the ssh keys or something.

DDs can now upload packages via ssh (scp, sftp, rsync+ssh) to
ssh.security.upload.debian.org[1].

  [1] 

Depending on how the uploads happen, they can remain readable by other
DDs for a short while (see the bugs mentioned in the announcement).  But
this might be less bad than uploading via unencrypted ftp.

DDs can also see which packages were uploaded as debianqueue's logs are
readable by local users, but I believe this shouldn't be too large of a
problem.

I'm not sure if buildds are already configured to upload to the security
archive via ssh as they do for the main archive.  It might be a good
idea to do so.

> - Use ftps (ftp over ssl), but I'm not sure how good that is
>   supported.
> - Encrypt the thing that is uploaded, then still use ftp.
>   We'd probably need a tool like debsign that puts it right
>   format.

I don't think we should use ftp for new services.

> - Some upload mechanism over https

This might still be useful as an alternative to ftp for DMs, also for
uploads to the main archive.  But DMs currently can't upload to the
security archive anyway (the DM permissions aren't synced to
security-master).

Ansgar

Bug#840104: Encrypted uploads to the security archive

[Kurt Roeckx]

Package: ftp.debian.org,security.debian.org

It's been a longstanding problem that the uploads to the security
archive are not encrypted in any way. I think this is a problem
for all embargoed uploads that we are doing.

Upstream might actually do all that's possible to keep the
security issues secret. But it can potentionally leak when it gets
uploaded to the security archive. As far as I know only ftp is
currently supported.

I can think of several ways of doing this, but you probably want
to talk to DSA about some of those options. They include:
- Allow uploads over ssh / sftp. This could be anonymous, or give
  access to the same user with all the ssh keys or something.
- Use ftps (ftp over ssl), but I'm not sure how good that is
  supported.
- Encrypt the thing that is uploaded, then still use ftp.
  We'd probably need a tool like debsign that puts it right
  format.
- Some upload mechanism over https


Kurt