Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Mojca Miklavec]

On 8 January 2016 at 01:47, Ryan Schmidt wrote:
> On Jan 7, 2016, at 12:12 PM, Eric A. Borisch wrote:
>
>> Bringing this back to the original point, it looks like there was some
>> discussion (I think) in this thread of making alocation (possibly
>> integrated in terms of access accounts with svn commit access)
>> available to store a 'snapshot' of a 'distfile' for instances like
>> this (where the source performs nightly refreshes of the tarball
>> without version information in the name.)
>>
>> So is that going to happen (soon), or is the desire I bake something
>> else up (drafting off curl-ca-bundle or graphviz-devel) for now?
>
> I have no plans to implement an arbitrary-file-hosting infrastructure for 
> MacPorts in the near future, and lots of other things I need to be working 
> on. You should solve the problem in the portfile.

One can put a file basically *anywhere* on the web, wait until it gets
mirrored and then safely remove both the file and the URL. Users would
then get the file from MacPorts mirror.

In the case of py-htmldocs you could probably commit a Portfile with
correct checksums for that day and the corresponding subdir (maybe
with a date if the filename doesn't change). Once the file gets
mirrored, just remove the URL of the file and make sure that all users
will get the file from the MacPorts mirror from that moment on. Once
you decide to switch to a newer version, repeat the process (uncomment
the URL, correct the checksums, commit, wait for the mirror, comment
out the URL, commit).

Mojca
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Russell Jones]

Another solution would be a PR with a patch for 
https://github.com/python/docsbuild-scripts/blob/master/build_docs.py to 
add versions, I guess.


Russell

On 08/01/16 09:11, Mojca Miklavec wrote:

On 8 January 2016 at 01:47, Ryan Schmidt wrote:

On Jan 7, 2016, at 12:12 PM, Eric A. Borisch wrote:


Bringing this back to the original point, it looks like there was some
discussion (I think) in this thread of making alocation (possibly
integrated in terms of access accounts with svn commit access)
available to store a 'snapshot' of a 'distfile' for instances like
this (where the source performs nightly refreshes of the tarball
without version information in the name.)

So is that going to happen (soon), or is the desire I bake something
else up (drafting off curl-ca-bundle or graphviz-devel) for now?

I have no plans to implement an arbitrary-file-hosting infrastructure for 
MacPorts in the near future, and lots of other things I need to be working on. 
You should solve the problem in the portfile.

One can put a file basically *anywhere* on the web, wait until it gets
mirrored and then safely remove both the file and the URL. Users would
then get the file from MacPorts mirror.

In the case of py-htmldocs you could probably commit a Portfile with
correct checksums for that day and the corresponding subdir (maybe
with a date if the filename doesn't change). Once the file gets
mirrored, just remove the URL of the file and make sure that all users
will get the file from the MacPorts mirror from that moment on. Once
you decide to switch to a newer version, repeat the process (uncomment
the URL, correct the checksums, commit, wait for the mirror, comment
out the URL, commit).

Mojca
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev


___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Joshua Root]

On 2016-1-8 05:12 , Eric A. Borisch wrote:
> Bringing this back to the original point, it looks like there was some
> discussion (I think) in this thread of making alocation (possibly
> integrated in terms of access accounts with svn commit access)
> available to store a 'snapshot' of a 'distfile' for instances like
> this (where the source performs nightly refreshes of the tarball
> without version information in the name.)
> 
> So is that going to happen (soon), or is the desire I bake something
> else up (drafting off curl-ca-bundle or graphviz-devel) for now?

I wonder if sourceforge's rules would allow making another project
called something like "macports-distfiles" and giving all our committers
the ability to add files to it. From a quick look at the terms of use,
it appears that uploaded content just needs to have a license "compliant
with the Open Source Initiative (“OSI”)’s Open Source Definition".

- Josh
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Ryan Schmidt]

On Jan 7, 2016, at 12:12 PM, Eric A. Borisch wrote:

> Bringing this back to the original point, it looks like there was some
> discussion (I think) in this thread of making alocation (possibly
> integrated in terms of access accounts with svn commit access)
> available to store a 'snapshot' of a 'distfile' for instances like
> this (where the source performs nightly refreshes of the tarball
> without version information in the name.)
> 
> So is that going to happen (soon), or is the desire I bake something
> else up (drafting off curl-ca-bundle or graphviz-devel) for now?


I have no plans to implement an arbitrary-file-hosting infrastructure for 
MacPorts in the near future, and lots of other things I need to be working on. 
You should solve the problem in the portfile.

___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Russell Jones]

On 06/01/16 23:44, Ryan Schmidt wrote:

On Jan 6, 2016, at 4:44 AM, Russell Jones wrote:


I was thinking you might use git+https://github.com/python/cpython.git/Doc with 
a set checkout id using the GitHub PortGroup, but that would require building 
the docs.

How about using https://docs.python.org and relying on python.org's SSL cert to 
ensure the integrity rather than the MacPorts checksum?

An SSL certificate does not guarantee the user is getting the same files the 
maintainer did. It only guarantees the user is talking to the same server. The 
server could be compromised, or (as is the case here) the developers could 
issue stealth updates.

Sure. It's just better than using http at making an MITM attack harder 
(though not impossible, as Daniel points out), which was the original 
objection. Better to do it right, though, definitely.


On Daniel's point: checking an SSL cert provides a guarantee from some 
certificate issuer, given a competent sysadmin, etc, that the host name 
matches it. Do you have some reason to think there are issuers in the 
root certificate list that would issue bogus python.org certs? Or are 
you talking about a cert being stolen? I'm not sure what you mean by 
"just ... valid".


Russell
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Russell Jones]

On 07/01/16 16:17, Daniel J. Luke wrote:

On Jan 7, 2016, at 5:53 AM, Russell Jones  
wrote:

On Daniel's point: checking an SSL cert provides a guarantee from some 
certificate issuer, given a competent sysadmin, etc, that the host name matches 
it.

When you validate an SSL certificate all you end up with is the assurance that 
some Certificate Authority has issued a certificate for that hostname.

That seems to me to be understating the case.

There are lots of CAs and they aren't immune to process (or other) issues (see 
also DigiNotar). There's a reason why there has been interest in public key 
pinning (and DANE + DNSSEC) - so you end up with a greater assurance.
Yes, and there was Lavabit, of course. But these are rather unusual 
cases. It's fine, important indeed, to exclude them as far as possible 
by ensuring oversight, but the problem still exists even with checksums, 
realistically.

Do you have some reason to think there are issuers in the root certificate list that 
would issue bogus python.org certs? Or are you talking about a cert being stolen? I'm not 
sure what you mean by "just .. valid".

I don't have reason to believe either of those things is currently happening - 
but I have reason to believe either is possible, and we shouldn't decide to 
rely on neither happening.
I agree. I wasn't suggesting we should in general. Rather, it was a 
suggestion for mitigation until a better one came along (which it did, 
very quickly).

Even in the non-malicious case, a re-org of files on python.org would yield 
unknown behavior (the file at that url could change, and in the base case we 
would get an error - in the worst case anything could be in that file).

Well no, not *anything* unless python.org to open it to the public, 
which seems a little far-fetched.


Russell
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Daniel J. Luke]

On Jan 7, 2016, at 5:53 AM, Russell Jones  
wrote:
> On Daniel's point: checking an SSL cert provides a guarantee from some 
> certificate issuer, given a competent sysadmin, etc, that the host name 
> matches it.

When you validate an SSL certificate all you end up with is the assurance that 
some Certificate Authority has issued a certificate for that hostname.

There are lots of CAs and they aren't immune to process (or other) issues (see 
also DigiNotar). There's a reason why there has been interest in public key 
pinning (and DANE + DNSSEC) - so you end up with a greater assurance.

> Do you have some reason to think there are issuers in the root certificate 
> list that would issue bogus python.org certs? Or are you talking about a cert 
> being stolen? I'm not sure what you mean by "just ... valid".

I don't have reason to believe either of those things is currently happening - 
but I have reason to believe either is possible, and we shouldn't decide to 
rely on neither happening.

Even in the non-malicious case, a re-org of files on python.org would yield 
unknown behavior (the file at that url could change, and in the base case we 
would get an error - in the worst case anything could be in that file).

-- 
Daniel J. Luke  
 
++ 
| * dl...@geeklair.net * |  

| *-- http://www.geeklair.net -* |  

++ 
|   Opinions expressed are mine and do not necessarily   |  

|  reflect the opinions of my employer.  |  

++





___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: SSL Certificates (was Re: [144262] trunk/dports/lang/py-htmldocs/Portfile)

[Russell Jones]

On 07/01/16 16:44, Daniel J. Luke wrote:

On Jan 7, 2016, at 11:41 AM, Russell Jones  
wrote:

On 07/01/16 16:17, Daniel J. Luke wrote:

On Jan 7, 2016, at 5:53 AM, Russell Jones  
wrote:

On Daniel's point: checking an SSL cert provides a guarantee from some 
certificate issuer, given a competent sysadmin, etc, that the host name matches 
it.

When you validate an SSL certificate all you end up with is the assurance that 
some Certificate Authority has issued a certificate for that hostname.

That seems to me to be understating the case.

How about a rephrase:

Validating an SSL certificate gives you strong cryptographic proof that a weak 
human process has occurred.

Fair enough, but of course it's relied on for distributing a base 
install of macports itself, along with the port files.

There are lots of CAs and they aren't immune to process (or other) issues (see 
also DigiNotar). There's a reason why there has been interest in public key 
pinning (and DANE + DNSSEC) - so you end up with a greater assurance.

Yes, and there was Lavabit, of course. But these are rather unusual cases. It's 
fine, important indeed, to exclude them as far as possible by ensuring 
oversight, but the problem still exists even with checksums, realistically.

Using a checksum is different. It gives you a guarantee that the file you 
downloaded was the same one that the maintainer intended the portfile to 
operate on.


Yes, and the maintainer can be fooled too.

Anyway, let's take this off list if you'd like to continue the conversation.

Russell
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Clemens Lang]

Hi,

On Thu, Jan 07, 2016 at 12:12:20PM -0600, Eric A. Borisch wrote:
> Bringing this back to the original point, it looks like there was some
> discussion (I think) in this thread of making alocation (possibly
> integrated in terms of access accounts with svn commit access)
> available to store a 'snapshot' of a 'distfile' for instances like
> this (where the source performs nightly refreshes of the tarball
> without version information in the name.)
> 
> So is that going to happen (soon), or is the desire I bake something
> else up (drafting off curl-ca-bundle or graphviz-devel) for now?

Maybe we could do something like user-directories on a webserver, where
every dev could put his stuff as needed? E.g. users.macports.org/~cal
for me?

I'm not sure how easy it would be to re-use authentication data, though.
If all else fails, we could just define a common location in the /users
folder where SSH public keys could be put and a post-commit hook would
fetch them and verify they've been committed by said user?

Of course there nothing stopping anybody with a webserver to implement
this, either.

But maybe it's just simpler to get storage at some cloud artifact
storage like Homebrew does for their archives, iirc.

-- 
Clemens
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Russell Jones]

I was thinking you might use 
git+https://github.com/python/cpython.git/Doc with a set checkout id 
using the GitHub PortGroup, but that would require building the docs.


How about using https://docs.python.org and relying on python.org's SSL 
cert to ensure the integrity rather than the MacPorts checksum?


Russell

On 05/01/16 20:10, Eric A. Borisch wrote:

On Tue, Jan 5, 2016 at 1:03 PM, Clemens Lang  wrote:

Hi,

On Tue, Jan 05, 2016 at 12:44:49PM -0600, Ryan Schmidt wrote:

I'm not comfortable with installing unchecked files on user systems.
The whole point of the checksum system is to verify that the files
that are installed on user systems are the same files that were tested
by the maintainer. By skipping the checksum phase you remove that
safeguard.

I agree, please revert that. This is an invitation for attackers.

I'd prefer to avoid it, obviously. Suggestions, then?

This is a tarball of documentation coming directly from
docs.python.org, which seem to be regenerated nightly (with new
checksums). The other versions are available in a stable form
(http://www.python.org/ftp/python/doc) but not the latest for 34
(3.4.4) or 35 (3.5.1). If there is another, stable, download location,
I'd be happy to point to it.

Thanks,
   - Eric
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev


___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Daniel J. Luke]

The general solution to this problem is to create a stable (snapshot) mirror 
somewhere else.

> On Jan 6, 2016, at 5:44 AM, Russell Jones  
> wrote:
> I was thinking you might use git+https://github.com/python/cpython.git/Doc 
> with a set checkout id using the GitHub PortGroup, but that would require 
> building the docs.
> 
> How about using https://docs.python.org and relying on python.org's SSL cert 
> to ensure the integrity rather than the MacPorts checksum?
> 
> Russell
> 
> On 05/01/16 20:10, Eric A. Borisch wrote:
>> On Tue, Jan 5, 2016 at 1:03 PM, Clemens Lang  wrote:
>>> Hi,
>>> 
>>> On Tue, Jan 05, 2016 at 12:44:49PM -0600, Ryan Schmidt wrote:
 I'm not comfortable with installing unchecked files on user systems.
 The whole point of the checksum system is to verify that the files
 that are installed on user systems are the same files that were tested
 by the maintainer. By skipping the checksum phase you remove that
 safeguard.
>>> I agree, please revert that. This is an invitation for attackers.
>> I'd prefer to avoid it, obviously. Suggestions, then?
>> 
>> This is a tarball of documentation coming directly from
>> docs.python.org, which seem to be regenerated nightly (with new
>> checksums). The other versions are available in a stable form
>> (http://www.python.org/ftp/python/doc) but not the latest for 34
>> (3.4.4) or 35 (3.5.1). If there is another, stable, download location,
>> I'd be happy to point to it.
>> 
>> Thanks,
>>   - Eric

-- 
Daniel J. Luke  
 
++ 
| * dl...@geeklair.net * |  

| *-- http://www.geeklair.net -* |  

++ 
|   Opinions expressed are mine and do not necessarily   |  

|  reflect the opinions of my employer.  |  

++





___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Ryan Schmidt]

On Jan 6, 2016, at 4:44 AM, Russell Jones wrote:

> I was thinking you might use git+https://github.com/python/cpython.git/Doc 
> with a set checkout id using the GitHub PortGroup, but that would require 
> building the docs.
> 
> How about using https://docs.python.org and relying on python.org's SSL cert 
> to ensure the integrity rather than the MacPorts checksum?

An SSL certificate does not guarantee the user is getting the same files the 
maintainer did. It only guarantees the user is talking to the same server. The 
server could be compromised, or (as is the case here) the developers could 
issue stealth updates.


One solution is to let the MacPorts distfiles mirror mirror the file, then 
switch the portfile to only look at the distfiles mirror, not the original 
server. This would need to be done every time you update the port. See the 
history of the graphviz-devel port for an example of this; their automated 
tarball generation system was recently changed and it now sometimes 
inadvertently repackages the current version with a stealth update. If this is 
going to happen often, as seems to be the case with py-htmldocs, it can be 
automated in the Portfile, to a degree. See the curl-ca-bundle subport of the 
curl port for an example of that. 

The ideal would be to work with the developers to convince them not to issue 
stealth updates.


___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Daniel J. Luke]

On Jan 6, 2016, at 6:44 PM, Ryan Schmidt  wrote:
> An SSL certificate does not guarantee the user is getting the same files the 
> maintainer did. It only guarantees the user is talking to the same server.

it's not even that strong of a guarantee (especially since the recommendation 
here was seemingly to just verify that the certificate is 'valid').

> One solution is to let the MacPorts distfiles mirror mirror the file, then 
> switch the portfile to only look at the distfiles mirror, not the original 
> server. This would need to be done every time you update the port.

Can we make it easier for maintainers to add files to the mirrors? When we used 
to put files into subversion, it was easy for any maintainer to avoid this 
problem by just checking in a snapshot. While it's undesirable to go back to 
that (storing lots of binaries in our svn repo isn't a great idea), being able 
to upload a snapshot again would be welcome.

It would fix this and to some extent also make it less 'necessary' for people 
to have ports fetching from source control systems (giving everyone the benefit 
of having the files mirrored and cacheable).

> The ideal would be to work with the developers to convince them not to issue 
> stealth updates.

+1 for this as well.

-- 
Daniel J. Luke  
 
++ 
| * dl...@geeklair.net * |  

| *-- http://www.geeklair.net -* |  

++ 
|   Opinions expressed are mine and do not necessarily   |  

|  reflect the opinions of my employer.  |  

++





___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: uploading distfiles directly to the distfiles mirror (was: Re: [144262] trunk/dports/lang/py-htmldocs/Portfile)

[Daniel J. Luke]

On Jan 6, 2016, at 8:35 PM, Ryan Schmidt  wrote:
> How would you envision this capability being provided? I wouldn't want to, 
> for example, just open up ftp write access to the distfiles server to anyone 
> who asked for it.

maybe for maintainers (or committers only)?

and probably 'sftp' instead of 'ftp'

> Would we make a web page where people can upload a distfile for a particular 
> port?

sure, that would work too.

I would think people's mirrors would go in a user directory too (similar to how 
the svn stuff worked) - so it would be obvious that it was a 
maintainer/committer sourced distfile.

> Would we make a new port command to submit a distfile?

that could work too (and might be a nice enhancement later to make it easier on 
maintainers) - but I don't think we need to wait for our (long) dev/release 
cycle to get something working.

> I don't really want to be in the business of distributing our own files. The 
> purpose of the mirror is to mirror files the developers provide elsewhere.

Sure, in the general case we shouldn't do it - but when the developer doesn't 
provide a good stable file (and/or only provides source from a source control 
system), then the best bad option is for us to create our own distfile.

-- 
Daniel J. Luke  
 
++ 
| * dl...@geeklair.net * |  

| *-- http://www.geeklair.net -* |  

++ 
|   Opinions expressed are mine and do not necessarily   |  

|  reflect the opinions of my employer.  |  

++





___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: uploading distfiles directly to the distfiles mirror (was: Re: [144262] trunk/dports/lang/py-htmldocs/Portfile)

[Ryan Schmidt]

On Jan 6, 2016, at 7:29 PM, Daniel J. Luke wrote:
> 
> Can we make it easier for maintainers to add files to the mirrors? When we 
> used to put files into subversion, it was easy for any maintainer to avoid 
> this problem by just checking in a snapshot. While it's undesirable to go 
> back to that (storing lots of binaries in our svn repo isn't a great idea), 
> being able to upload a snapshot again would be welcome.
> 
> It would fix this and to some extent also make it less 'necessary' for people 
> to have ports fetching from source control systems (giving everyone the 
> benefit of having the files mirrored and cacheable).

How would you envision this capability being provided? I wouldn't want to, for 
example, just open up ftp write access to the distfiles server to anyone who 
asked for it.

Would we make a web page where people can upload a distfile for a particular 
port?

Would we make a new port command to submit a distfile?


I don't really want to be in the business of distributing our own files. The 
purpose of the mirror is to mirror files the developers provide elsewhere.

___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Clemens Lang]

Hi,

On Tue, Jan 05, 2016 at 12:44:49PM -0600, Ryan Schmidt wrote:
> 
> I'm not comfortable with installing unchecked files on user systems.
> The whole point of the checksum system is to verify that the files
> that are installed on user systems are the same files that were tested
> by the maintainer. By skipping the checksum phase you remove that
> safeguard.

I agree, please revert that. This is an invitation for attackers.

-- 
Clemens
___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev

Re: [144262] trunk/dports/lang/py-htmldocs/Portfile

[Ryan Schmidt]

> On Jan 5, 2016, at 12:43 PM, ebori...@macports.org wrote:
> 
> Revision
> 144262
> Author
> ebori...@macports.org
> Date
> 2016-01-05 10:43:29 -0800 (Tue, 05 Jan 2016)
> Log Message
> 
> py-htmldocs: 34 and 35 tarballs appear to be updated nightly; skip checksums. 
> (At least we don't install any execs/libs.) Move dist_subdir line so it grabs 
> correct revision.
> Modified Paths
>   • trunk/dports/lang/py-htmldocs/Portfile

>  if {${python.version} == 34} {
>master_sites  http://docs.python.org/${python.branch}/archives
> -  checksums \
> -rmd160  61a2f71b85b5e50c38de8e5616ffd87a4a382bf6 \
> -sha256  
> 8554b59a5aea9801ae735cafd13585f6e07c25cb71dba2402bd4eacfc05ee2e0
> +  # These seem to be re-generated daily.
> +  checksum {}
>  }
>  
>  if {${python.version} == 35} {
>master_sites  http://docs.python.org/${python.branch}/archives
>set revision  [expr ${base_rev}+1] 
> -  checksums \
> -rmd160  3e99602293474b478c97ee7ba004065ab65d9196 \
> -sha256  
> 1e395e25837fd8898cdd56511c6b56a2ceaace3145e2913e3544a4dfa0f63916
> +  # These seem to be re-generated daily.
> +  checksum {}

I'm not comfortable with installing unchecked files on user systems. The whole 
point of the checksum system is to verify that the files that are installed on 
user systems are the same files that were tested by the maintainer. By skipping 
the checksum phase you remove that safeguard.


___
macports-dev mailing list
macports-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/macports-dev