Re: Extend closer.lua to act as a download page

[sebb]

On Mon, 18 Apr 2022 at 04:28, Craig Russell  wrote:
>
>
>
> On Apr 17, 2022, at 4:02 PM, sebb  wrote:
>
> On Sun, 17 Apr 2022 at 23:31, Christopher  wrote:
>
>
> I don't think this is proposing to have links to the
> checksums/signatures on the CDN, but rather to have the closer.lua
> template, which is still controlled by INFRA, include links to those
> files at the recommended location at downloads.apache.org. closer.lua
> can easily link to $x at dlcdn.apache.org, but $x.sha512 and $x.asc
> and KEYS at downloads.apache.org.
>
>
> Yes, the idea is to extend the existing page to create a minimal valid
> download page for projects to use.
> The links to KEYS, sigs and hashes would use the same URLs as a
> regular download page
>
>
> Finding the .asc and either .sha256 or .sha512 (or both) is relatively 
> straightforward. As Sebb noted earlier, finding the KEYS file may be a bit 
> harder.
>
>
> FTR, I have got a working Docker image which I will upload shortly.
>

https://github.com/apache/infrastructure-p6/tree/sebb/closer

Clone the repo branch
cd modules/closer_cgi
Read comments at end of Dockerfile

> Looking forward to seeing the patch to closer.lua and closer.html.
>
> Craig
>
>
> On Sun, Apr 17, 2022 at 5:46 PM Jarek Potiuk  wrote:
>
>
> Sebb,
>
> I think it would be great to see some result analysis on
> security/integrity of the proposal. I wonder, did you make any
> analysis on the integrity aspect? What's the result of it?
>
> I believe originally the requirement for separating file downloads and
> sha/checksum were to make sure that the checksum/sha files are always
> served directly from the fully apache-controlled delivery chain.
> Previously, when there were mirrors, this was the only way to make
> sure you could validate the provenience of the files - the .sha and
> .sum files were always served from "https://download.a.o"; so - unless
> your local certificate authority source were compromised, you could be
> sure that the downloaded files were not tampered with. It's all too
> easy to modify both - the file and their checksum/sha to "pretend it
> is valid".
>
> I understand the main reason why this was done this way was that
> directly the Apache-owned and fully controlled servers were able to
> serve all those files - because they are very small and can be served
> directly by the Apache server under the infra control, while using
> mirrors/cdn is absolutely necessary to serve the binary content which
> is huge, and we cannot really route all the users to hit directly the
> apache download server. This is at least  when I tried to understand
> the context and reasoning why it was implemented this way.
>
> I am not sure how dlcdn works and how "trustful" it is for the Apache
> to be 100% sure the content has not been tampered with by the CDN
> provider.
> Do you know what is the consequence of your proposal on the integrity
> check in case they are part of close.lua? Does it mean that it will be
> served from a "fully owned" apache page ? Is it something that ASF can
> "delegate" to the CDN provider? Is it something that closer.lua can
> handle in the "provenience sure way" ? Is the content served  by
> closer.lua fully served by the ASF-owned server and cannot be tampered
> with by the CDN provider?
>
> I do not know the answer to those questions (I do not know too much
> about the closer.lua scripts and what's the level of trust between the
> ASF and CDN) - but I am sure that when you proposed the changes, you
> considered the context and reasoning why it was separated - so can you
> tell us all if all the integrity and "provenience" aspects are handled
> well by your proposal?
>
> J.
>
> On Sun, Apr 17, 2022 at 1:06 PM sebb  wrote:
>
>
> On Sun, 17 Apr 2022 at 11:56, sebb  wrote:
>
>
> On Sun, 17 Apr 2022 at 10:36, Greg Stein  wrote:
>
>
> Hi Craig,
>
> The closer.lua, its prior name closer.cgi. and all predecessors have been 
> fine with the Foundation's consumers liking its functionality. This has 
> supported 200+ projects (or more, depending on how you'd like to count). 
> Changes to that system are (thus) demand-based, which we have not seen. 
> Please provide some PRs with the changes you would like to make.
>
> I do believe that others agree with your ideas on "automated hashes", though 
> I really have no idea on the follow-on security details. Nor a mechanism for 
> broad-based security chains of GPG keys and hashes for the artifacts.
>
>
> My idea here was just to link to the sig and hash by adding the
> appropriate suffix to the artifact path name, and checking that it
> exists.
> e.g. look for db/jdo/3.2/jdo-3.2-source-release.tar.gz + .asc also
> .sha256 or .sha512 etc.
> The code already checks for the presence of the artifact.
>
> There aren't that many valid hash types to check; if the code cannot
> find the required files, it could direct users back to the project.
>
> Figure it out, discuss on the mailing list, file some Pull-Requests, and we 
> can move forward.
>
>

Re: Extend closer.lua to act as a download page

[Craig Russell]

> On Apr 17, 2022, at 4:02 PM, sebb  wrote:
> 
> On Sun, 17 Apr 2022 at 23:31, Christopher  > wrote:
>> 
>> I don't think this is proposing to have links to the
>> checksums/signatures on the CDN, but rather to have the closer.lua
>> template, which is still controlled by INFRA, include links to those
>> files at the recommended location at downloads.apache.org. closer.lua
>> can easily link to $x at dlcdn.apache.org, but $x.sha512 and $x.asc
>> and KEYS at downloads.apache.org.
> 
> Yes, the idea is to extend the existing page to create a minimal valid
> download page for projects to use.
> The links to KEYS, sigs and hashes would use the same URLs as a
> regular download page

Finding the .asc and either .sha256 or .sha512 (or both) is relatively 
straightforward. As Sebb noted earlier, finding the KEYS file may be a bit 
harder.
> 
> FTR, I have got a working Docker image which I will upload shortly.

Looking forward to seeing the patch to closer.lua and closer.html.

Craig
> 
>> On Sun, Apr 17, 2022 at 5:46 PM Jarek Potiuk  wrote:
>>> 
>>> Sebb,
>>> 
>>> I think it would be great to see some result analysis on
>>> security/integrity of the proposal. I wonder, did you make any
>>> analysis on the integrity aspect? What's the result of it?
>>> 
>>> I believe originally the requirement for separating file downloads and
>>> sha/checksum were to make sure that the checksum/sha files are always
>>> served directly from the fully apache-controlled delivery chain.
>>> Previously, when there were mirrors, this was the only way to make
>>> sure you could validate the provenience of the files - the .sha and
>>> .sum files were always served from "https://download.a.o"; so - unless
>>> your local certificate authority source were compromised, you could be
>>> sure that the downloaded files were not tampered with. It's all too
>>> easy to modify both - the file and their checksum/sha to "pretend it
>>> is valid".
>>> 
>>> I understand the main reason why this was done this way was that
>>> directly the Apache-owned and fully controlled servers were able to
>>> serve all those files - because they are very small and can be served
>>> directly by the Apache server under the infra control, while using
>>> mirrors/cdn is absolutely necessary to serve the binary content which
>>> is huge, and we cannot really route all the users to hit directly the
>>> apache download server. This is at least  when I tried to understand
>>> the context and reasoning why it was implemented this way.
>>> 
>>> I am not sure how dlcdn works and how "trustful" it is for the Apache
>>> to be 100% sure the content has not been tampered with by the CDN
>>> provider.
>>> Do you know what is the consequence of your proposal on the integrity
>>> check in case they are part of close.lua? Does it mean that it will be
>>> served from a "fully owned" apache page ? Is it something that ASF can
>>> "delegate" to the CDN provider? Is it something that closer.lua can
>>> handle in the "provenience sure way" ? Is the content served  by
>>> closer.lua fully served by the ASF-owned server and cannot be tampered
>>> with by the CDN provider?
>>> 
>>> I do not know the answer to those questions (I do not know too much
>>> about the closer.lua scripts and what's the level of trust between the
>>> ASF and CDN) - but I am sure that when you proposed the changes, you
>>> considered the context and reasoning why it was separated - so can you
>>> tell us all if all the integrity and "provenience" aspects are handled
>>> well by your proposal?
>>> 
>>> J.
>>> 
>>> On Sun, Apr 17, 2022 at 1:06 PM sebb  wrote:
 
 On Sun, 17 Apr 2022 at 11:56, sebb  wrote:
> 
> On Sun, 17 Apr 2022 at 10:36, Greg Stein  wrote:
>> 
>> Hi Craig,
>> 
>> The closer.lua, its prior name closer.cgi. and all predecessors have 
>> been fine with the Foundation's consumers liking its functionality. This 
>> has supported 200+ projects (or more, depending on how you'd like to 
>> count). Changes to that system are (thus) demand-based, which we have 
>> not seen. Please provide some PRs with the changes you would like to 
>> make.
>> 
>> I do believe that others agree with your ideas on "automated hashes", 
>> though I really have no idea on the follow-on security details. Nor a 
>> mechanism for broad-based security chains of GPG keys and hashes for the 
>> artifacts.
> 
> My idea here was just to link to the sig and hash by adding the
> appropriate suffix to the artifact path name, and checking that it
> exists.
> e.g. look for db/jdo/3.2/jdo-3.2-source-release.tar.gz + .asc also
> .sha256 or .sha512 etc.
> The code already checks for the presence of the artifact.
> 
> There aren't that many valid hash types to check; if the code cannot
> find the required files, it could direct users back to the project.
> 
>> Figure it out, discuss on the mailing l

Re: Extend closer.lua to act as a download page

[sebb]

On Sun, 17 Apr 2022 at 23:31, Christopher  wrote:
>
> I don't think this is proposing to have links to the
> checksums/signatures on the CDN, but rather to have the closer.lua
> template, which is still controlled by INFRA, include links to those
> files at the recommended location at downloads.apache.org. closer.lua
> can easily link to $x at dlcdn.apache.org, but $x.sha512 and $x.asc
> and KEYS at downloads.apache.org.

Yes, the idea is to extend the existing page to create a minimal valid
download page for projects to use.
The links to KEYS, sigs and hashes would use the same URLs as a
regular download page

FTR, I have got a working Docker image which I will upload shortly.

> On Sun, Apr 17, 2022 at 5:46 PM Jarek Potiuk  wrote:
> >
> > Sebb,
> >
> > I think it would be great to see some result analysis on
> > security/integrity of the proposal. I wonder, did you make any
> > analysis on the integrity aspect? What's the result of it?
> >
> > I believe originally the requirement for separating file downloads and
> > sha/checksum were to make sure that the checksum/sha files are always
> > served directly from the fully apache-controlled delivery chain.
> > Previously, when there were mirrors, this was the only way to make
> > sure you could validate the provenience of the files - the .sha and
> > .sum files were always served from "https://download.a.o"; so - unless
> > your local certificate authority source were compromised, you could be
> > sure that the downloaded files were not tampered with. It's all too
> > easy to modify both - the file and their checksum/sha to "pretend it
> > is valid".
> >
> > I understand the main reason why this was done this way was that
> > directly the Apache-owned and fully controlled servers were able to
> > serve all those files - because they are very small and can be served
> > directly by the Apache server under the infra control, while using
> > mirrors/cdn is absolutely necessary to serve the binary content which
> > is huge, and we cannot really route all the users to hit directly the
> > apache download server. This is at least  when I tried to understand
> > the context and reasoning why it was implemented this way.
> >
> > I am not sure how dlcdn works and how "trustful" it is for the Apache
> > to be 100% sure the content has not been tampered with by the CDN
> > provider.
> > Do you know what is the consequence of your proposal on the integrity
> > check in case they are part of close.lua? Does it mean that it will be
> > served from a "fully owned" apache page ? Is it something that ASF can
> > "delegate" to the CDN provider? Is it something that closer.lua can
> > handle in the "provenience sure way" ? Is the content served  by
> > closer.lua fully served by the ASF-owned server and cannot be tampered
> > with by the CDN provider?
> >
> > I do not know the answer to those questions (I do not know too much
> > about the closer.lua scripts and what's the level of trust between the
> > ASF and CDN) - but I am sure that when you proposed the changes, you
> > considered the context and reasoning why it was separated - so can you
> > tell us all if all the integrity and "provenience" aspects are handled
> > well by your proposal?
> >
> > J.
> >
> > On Sun, Apr 17, 2022 at 1:06 PM sebb  wrote:
> > >
> > > On Sun, 17 Apr 2022 at 11:56, sebb  wrote:
> > > >
> > > > On Sun, 17 Apr 2022 at 10:36, Greg Stein  wrote:
> > > > >
> > > > > Hi Craig,
> > > > >
> > > > > The closer.lua, its prior name closer.cgi. and all predecessors have 
> > > > > been fine with the Foundation's consumers liking its functionality. 
> > > > > This has supported 200+ projects (or more, depending on how you'd 
> > > > > like to count). Changes to that system are (thus) demand-based, which 
> > > > > we have not seen. Please provide some PRs with the changes you would 
> > > > > like to make.
> > > > >
> > > > > I do believe that others agree with your ideas on "automated hashes", 
> > > > > though I really have no idea on the follow-on security details. Nor a 
> > > > > mechanism for broad-based security chains of GPG keys and hashes for 
> > > > > the artifacts.
> > > >
> > > > My idea here was just to link to the sig and hash by adding the
> > > > appropriate suffix to the artifact path name, and checking that it
> > > > exists.
> > > > e.g. look for db/jdo/3.2/jdo-3.2-source-release.tar.gz + .asc also
> > > > .sha256 or .sha512 etc.
> > > > The code already checks for the presence of the artifact.
> > > >
> > > > There aren't that many valid hash types to check; if the code cannot
> > > > find the required files, it could direct users back to the project.
> > > >
> > > > > Figure it out, discuss on the mailing list, file some Pull-Requests, 
> > > > > and we can move forward.
> > >
> > > It looks like the code is currently at:
> > >
> > > https://github.com/apache/infrastructure-p6/blob/production/modules/closer_cgi/files/closer.lua
> > > (requires GitHub login linked to ASF login with a

Re: Extend closer.lua to act as a download page

[Christopher]

I don't think this is proposing to have links to the
checksums/signatures on the CDN, but rather to have the closer.lua
template, which is still controlled by INFRA, include links to those
files at the recommended location at downloads.apache.org. closer.lua
can easily link to $x at dlcdn.apache.org, but $x.sha512 and $x.asc
and KEYS at downloads.apache.org.

On Sun, Apr 17, 2022 at 5:46 PM Jarek Potiuk  wrote:
>
> Sebb,
>
> I think it would be great to see some result analysis on
> security/integrity of the proposal. I wonder, did you make any
> analysis on the integrity aspect? What's the result of it?
>
> I believe originally the requirement for separating file downloads and
> sha/checksum were to make sure that the checksum/sha files are always
> served directly from the fully apache-controlled delivery chain.
> Previously, when there were mirrors, this was the only way to make
> sure you could validate the provenience of the files - the .sha and
> .sum files were always served from "https://download.a.o"; so - unless
> your local certificate authority source were compromised, you could be
> sure that the downloaded files were not tampered with. It's all too
> easy to modify both - the file and their checksum/sha to "pretend it
> is valid".
>
> I understand the main reason why this was done this way was that
> directly the Apache-owned and fully controlled servers were able to
> serve all those files - because they are very small and can be served
> directly by the Apache server under the infra control, while using
> mirrors/cdn is absolutely necessary to serve the binary content which
> is huge, and we cannot really route all the users to hit directly the
> apache download server. This is at least  when I tried to understand
> the context and reasoning why it was implemented this way.
>
> I am not sure how dlcdn works and how "trustful" it is for the Apache
> to be 100% sure the content has not been tampered with by the CDN
> provider.
> Do you know what is the consequence of your proposal on the integrity
> check in case they are part of close.lua? Does it mean that it will be
> served from a "fully owned" apache page ? Is it something that ASF can
> "delegate" to the CDN provider? Is it something that closer.lua can
> handle in the "provenience sure way" ? Is the content served  by
> closer.lua fully served by the ASF-owned server and cannot be tampered
> with by the CDN provider?
>
> I do not know the answer to those questions (I do not know too much
> about the closer.lua scripts and what's the level of trust between the
> ASF and CDN) - but I am sure that when you proposed the changes, you
> considered the context and reasoning why it was separated - so can you
> tell us all if all the integrity and "provenience" aspects are handled
> well by your proposal?
>
> J.
>
> On Sun, Apr 17, 2022 at 1:06 PM sebb  wrote:
> >
> > On Sun, 17 Apr 2022 at 11:56, sebb  wrote:
> > >
> > > On Sun, 17 Apr 2022 at 10:36, Greg Stein  wrote:
> > > >
> > > > Hi Craig,
> > > >
> > > > The closer.lua, its prior name closer.cgi. and all predecessors have 
> > > > been fine with the Foundation's consumers liking its functionality. 
> > > > This has supported 200+ projects (or more, depending on how you'd like 
> > > > to count). Changes to that system are (thus) demand-based, which we 
> > > > have not seen. Please provide some PRs with the changes you would like 
> > > > to make.
> > > >
> > > > I do believe that others agree with your ideas on "automated hashes", 
> > > > though I really have no idea on the follow-on security details. Nor a 
> > > > mechanism for broad-based security chains of GPG keys and hashes for 
> > > > the artifacts.
> > >
> > > My idea here was just to link to the sig and hash by adding the
> > > appropriate suffix to the artifact path name, and checking that it
> > > exists.
> > > e.g. look for db/jdo/3.2/jdo-3.2-source-release.tar.gz + .asc also
> > > .sha256 or .sha512 etc.
> > > The code already checks for the presence of the artifact.
> > >
> > > There aren't that many valid hash types to check; if the code cannot
> > > find the required files, it could direct users back to the project.
> > >
> > > > Figure it out, discuss on the mailing list, file some Pull-Requests, 
> > > > and we can move forward.
> >
> > It looks like the code is currently at:
> >
> > https://github.com/apache/infrastructure-p6/blob/production/modules/closer_cgi/files/closer.lua
> > (requires GitHub login linked to ASF login with appropriate karma)
> >
> > > > Cheers,
> > > > Greg
> > > > InfraAdmin, ASF
> > > >
> > > >
> > > > On Sat, Apr 16, 2022 at 6:23 PM Craig Russell  
> > > > wrote:
> > > >>
> > > >> I'd like to ask on behalf of the DB PMC for the infra tool closer.lua 
> > > >> to be enhanced a bit for use by projects.
> > > >>
> > > >> It will be much easier for us if we can simply link to e.g.
> > > >> https://www.apache.org/dyn/closer.lua/db/jdo/3.2/jdo-3.2-source-release.tar.gz
> > > >> and have that page 

Re: Extend closer.lua to act as a download page

[Jarek Potiuk]

Sebb,

I think it would be great to see some result analysis on
security/integrity of the proposal. I wonder, did you make any
analysis on the integrity aspect? What's the result of it?

I believe originally the requirement for separating file downloads and
sha/checksum were to make sure that the checksum/sha files are always
served directly from the fully apache-controlled delivery chain.
Previously, when there were mirrors, this was the only way to make
sure you could validate the provenience of the files - the .sha and
.sum files were always served from "https://download.a.o"; so - unless
your local certificate authority source were compromised, you could be
sure that the downloaded files were not tampered with. It's all too
easy to modify both - the file and their checksum/sha to "pretend it
is valid".

I understand the main reason why this was done this way was that
directly the Apache-owned and fully controlled servers were able to
serve all those files - because they are very small and can be served
directly by the Apache server under the infra control, while using
mirrors/cdn is absolutely necessary to serve the binary content which
is huge, and we cannot really route all the users to hit directly the
apache download server. This is at least  when I tried to understand
the context and reasoning why it was implemented this way.

I am not sure how dlcdn works and how "trustful" it is for the Apache
to be 100% sure the content has not been tampered with by the CDN
provider.
Do you know what is the consequence of your proposal on the integrity
check in case they are part of close.lua? Does it mean that it will be
served from a "fully owned" apache page ? Is it something that ASF can
"delegate" to the CDN provider? Is it something that closer.lua can
handle in the "provenience sure way" ? Is the content served  by
closer.lua fully served by the ASF-owned server and cannot be tampered
with by the CDN provider?

I do not know the answer to those questions (I do not know too much
about the closer.lua scripts and what's the level of trust between the
ASF and CDN) - but I am sure that when you proposed the changes, you
considered the context and reasoning why it was separated - so can you
tell us all if all the integrity and "provenience" aspects are handled
well by your proposal?

J.

On Sun, Apr 17, 2022 at 1:06 PM sebb  wrote:
>
> On Sun, 17 Apr 2022 at 11:56, sebb  wrote:
> >
> > On Sun, 17 Apr 2022 at 10:36, Greg Stein  wrote:
> > >
> > > Hi Craig,
> > >
> > > The closer.lua, its prior name closer.cgi. and all predecessors have been 
> > > fine with the Foundation's consumers liking its functionality. This has 
> > > supported 200+ projects (or more, depending on how you'd like to count). 
> > > Changes to that system are (thus) demand-based, which we have not seen. 
> > > Please provide some PRs with the changes you would like to make.
> > >
> > > I do believe that others agree with your ideas on "automated hashes", 
> > > though I really have no idea on the follow-on security details. Nor a 
> > > mechanism for broad-based security chains of GPG keys and hashes for the 
> > > artifacts.
> >
> > My idea here was just to link to the sig and hash by adding the
> > appropriate suffix to the artifact path name, and checking that it
> > exists.
> > e.g. look for db/jdo/3.2/jdo-3.2-source-release.tar.gz + .asc also
> > .sha256 or .sha512 etc.
> > The code already checks for the presence of the artifact.
> >
> > There aren't that many valid hash types to check; if the code cannot
> > find the required files, it could direct users back to the project.
> >
> > > Figure it out, discuss on the mailing list, file some Pull-Requests, and 
> > > we can move forward.
>
> It looks like the code is currently at:
>
> https://github.com/apache/infrastructure-p6/blob/production/modules/closer_cgi/files/closer.lua
> (requires GitHub login linked to ASF login with appropriate karma)
>
> > > Cheers,
> > > Greg
> > > InfraAdmin, ASF
> > >
> > >
> > > On Sat, Apr 16, 2022 at 6:23 PM Craig Russell  
> > > wrote:
> > >>
> > >> I'd like to ask on behalf of the DB PMC for the infra tool closer.lua to 
> > >> be enhanced a bit for use by projects.
> > >>
> > >> It will be much easier for us if we can simply link to e.g.
> > >> https://www.apache.org/dyn/closer.lua/db/jdo/3.2/jdo-3.2-source-release.tar.gz
> > >> and have that page contain the .asc and .sha512 as well.
> > >>
> > >> Even better if closer.lua can also find the relevant KEYS file and link 
> > >> to it when describing how to check the release.
> > >>
> > >> With this approach, it is easier for the project and less error-prone to 
> > >> maintain the download page(s). We can include the current release as 
> > >> well as archived releases on the same download page and omit the links 
> > >> to the .asc and .sha512 and KEYS files.
> > >>
> > >> Please let us know how we can help.
> > >>
> > >> Regards,
> > >> Craig
> > >>
> > >>
> > >> On Apr 16, 2022, at 2:01 AM, Greg S

Re: Extend closer.lua to act as a download page

[sebb]

On Sun, 17 Apr 2022 at 11:56, sebb  wrote:
>
> On Sun, 17 Apr 2022 at 10:36, Greg Stein  wrote:
> >
> > Hi Craig,
> >
> > The closer.lua, its prior name closer.cgi. and all predecessors have been 
> > fine with the Foundation's consumers liking its functionality. This has 
> > supported 200+ projects (or more, depending on how you'd like to count). 
> > Changes to that system are (thus) demand-based, which we have not seen. 
> > Please provide some PRs with the changes you would like to make.
> >
> > I do believe that others agree with your ideas on "automated hashes", 
> > though I really have no idea on the follow-on security details. Nor a 
> > mechanism for broad-based security chains of GPG keys and hashes for the 
> > artifacts.
>
> My idea here was just to link to the sig and hash by adding the
> appropriate suffix to the artifact path name, and checking that it
> exists.
> e.g. look for db/jdo/3.2/jdo-3.2-source-release.tar.gz + .asc also
> .sha256 or .sha512 etc.
> The code already checks for the presence of the artifact.
>
> There aren't that many valid hash types to check; if the code cannot
> find the required files, it could direct users back to the project.
>
> > Figure it out, discuss on the mailing list, file some Pull-Requests, and we 
> > can move forward.

It looks like the code is currently at:

https://github.com/apache/infrastructure-p6/blob/production/modules/closer_cgi/files/closer.lua
(requires GitHub login linked to ASF login with appropriate karma)

> > Cheers,
> > Greg
> > InfraAdmin, ASF
> >
> >
> > On Sat, Apr 16, 2022 at 6:23 PM Craig Russell  wrote:
> >>
> >> I'd like to ask on behalf of the DB PMC for the infra tool closer.lua to 
> >> be enhanced a bit for use by projects.
> >>
> >> It will be much easier for us if we can simply link to e.g.
> >> https://www.apache.org/dyn/closer.lua/db/jdo/3.2/jdo-3.2-source-release.tar.gz
> >> and have that page contain the .asc and .sha512 as well.
> >>
> >> Even better if closer.lua can also find the relevant KEYS file and link to 
> >> it when describing how to check the release.
> >>
> >> With this approach, it is easier for the project and less error-prone to 
> >> maintain the download page(s). We can include the current release as well 
> >> as archived releases on the same download page and omit the links to the 
> >> .asc and .sha512 and KEYS files.
> >>
> >> Please let us know how we can help.
> >>
> >> Regards,
> >> Craig
> >>
> >>
> >> On Apr 16, 2022, at 2:01 AM, Greg Stein  wrote:
> >>
> >> On Thu, Apr 14, 2022 at 4:43 PM sebb  wrote:
> >>>
> >>> On Thu, 14 Apr 2022 at 20:52, Christopher  wrote:
> >>> >
> >>> > Doing this would probably restrict INFRA's ability to adapt it to
> >>> > their changing needs, as projects would become dependent on it.
> >>>
> >>> Projects are already dependent on the page.
> >>
> >>
> >> Correct. We *want* projects to use closer.lua. It provides a control point 
> >> to direct users to the appropriate location to download. Today, it 
> >> primarily sends them to our CDN and to an EU download location.
> >>
> >> Should those decisions ever change, we want projects to be using 
> >> closer.lua to effect those changes.
> >>
> >>>
> >>> I think it would make it easier for Infra to make changes, as they
> >>> would be able to adjust it.
> >>
> >>
> >> Yes.
> >>
> >>>
> >>> > Probably best to leave the mirror-management page independent from the
> >>> > project download pages, since they serve the needs of separate groups.
> >>>
> >>> I was not suggesting replacing these, for those projects that want to
> >>> customise the pages.
> >>
> >>
> >> Today, we no longer have a mirror system. But requiring projects to use 
> >> closer.lua ensures that we can swap in that future option.
> >>
> >> All that said, as I recall: closer.lua provides support for .ezt pages so 
> >> that projects can provide custom download pages. Maybe there is a way to 
> >> provide the needed variables to projects' custom download page templates.
> >>
> >> Or, they can just keep using their pages.
> >>
> >> The download pages are owned by the TLPs. Infra isn't gonna interfere with 
> >> them. Should any TLPs want more features, then they can ask. We haven't 
> >> seen any requests in years from the TLPs.
> >>
> >> Cheers,
> >> Greg
> >> InfraAdmin, ASF
> >>
> >>
> >>
> >> Craig L Russell
> >> c...@apache.org
> >>

Re: Extend closer.lua to act as a download page

[sebb]

On Sun, 17 Apr 2022 at 10:36, Greg Stein  wrote:
>
> Hi Craig,
>
> The closer.lua, its prior name closer.cgi. and all predecessors have been 
> fine with the Foundation's consumers liking its functionality. This has 
> supported 200+ projects (or more, depending on how you'd like to count). 
> Changes to that system are (thus) demand-based, which we have not seen. 
> Please provide some PRs with the changes you would like to make.
>
> I do believe that others agree with your ideas on "automated hashes", though 
> I really have no idea on the follow-on security details. Nor a mechanism for 
> broad-based security chains of GPG keys and hashes for the artifacts.

My idea here was just to link to the sig and hash by adding the
appropriate suffix to the artifact path name, and checking that it
exists.
e.g. look for db/jdo/3.2/jdo-3.2-source-release.tar.gz + .asc also
.sha256 or .sha512 etc.
The code already checks for the presence of the artifact.

There aren't that many valid hash types to check; if the code cannot
find the required files, it could direct users back to the project.

> Figure it out, discuss on the mailing list, file some Pull-Requests, and we 
> can move forward.

> Cheers,
> Greg
> InfraAdmin, ASF
>
>
> On Sat, Apr 16, 2022 at 6:23 PM Craig Russell  wrote:
>>
>> I'd like to ask on behalf of the DB PMC for the infra tool closer.lua to be 
>> enhanced a bit for use by projects.
>>
>> It will be much easier for us if we can simply link to e.g.
>> https://www.apache.org/dyn/closer.lua/db/jdo/3.2/jdo-3.2-source-release.tar.gz
>> and have that page contain the .asc and .sha512 as well.
>>
>> Even better if closer.lua can also find the relevant KEYS file and link to 
>> it when describing how to check the release.
>>
>> With this approach, it is easier for the project and less error-prone to 
>> maintain the download page(s). We can include the current release as well as 
>> archived releases on the same download page and omit the links to the .asc 
>> and .sha512 and KEYS files.
>>
>> Please let us know how we can help.
>>
>> Regards,
>> Craig
>>
>>
>> On Apr 16, 2022, at 2:01 AM, Greg Stein  wrote:
>>
>> On Thu, Apr 14, 2022 at 4:43 PM sebb  wrote:
>>>
>>> On Thu, 14 Apr 2022 at 20:52, Christopher  wrote:
>>> >
>>> > Doing this would probably restrict INFRA's ability to adapt it to
>>> > their changing needs, as projects would become dependent on it.
>>>
>>> Projects are already dependent on the page.
>>
>>
>> Correct. We *want* projects to use closer.lua. It provides a control point 
>> to direct users to the appropriate location to download. Today, it primarily 
>> sends them to our CDN and to an EU download location.
>>
>> Should those decisions ever change, we want projects to be using closer.lua 
>> to effect those changes.
>>
>>>
>>> I think it would make it easier for Infra to make changes, as they
>>> would be able to adjust it.
>>
>>
>> Yes.
>>
>>>
>>> > Probably best to leave the mirror-management page independent from the
>>> > project download pages, since they serve the needs of separate groups.
>>>
>>> I was not suggesting replacing these, for those projects that want to
>>> customise the pages.
>>
>>
>> Today, we no longer have a mirror system. But requiring projects to use 
>> closer.lua ensures that we can swap in that future option.
>>
>> All that said, as I recall: closer.lua provides support for .ezt pages so 
>> that projects can provide custom download pages. Maybe there is a way to 
>> provide the needed variables to projects' custom download page templates.
>>
>> Or, they can just keep using their pages.
>>
>> The download pages are owned by the TLPs. Infra isn't gonna interfere with 
>> them. Should any TLPs want more features, then they can ask. We haven't seen 
>> any requests in years from the TLPs.
>>
>> Cheers,
>> Greg
>> InfraAdmin, ASF
>>
>>
>>
>> Craig L Russell
>> c...@apache.org
>>

Re: Extend closer.lua to act as a download page

[Greg Stein]

Hi Craig,

The closer.lua, its prior name closer.cgi. and all predecessors have been
fine with the Foundation's consumers liking its functionality. This has
supported 200+ projects (or more, depending on how you'd like to count).
Changes to that system are (thus) demand-based, which we have not seen.
Please provide some PRs with the changes you would like to make.

I do believe that others agree with your ideas on "automated hashes",
though I really have no idea on the follow-on security details. Nor a
mechanism for broad-based security chains of GPG keys and hashes for the
artifacts.

Figure it out, discuss on the mailing list, file some Pull-Requests, and we
can move forward.

Cheers,
Greg
InfraAdmin, ASF


On Sat, Apr 16, 2022 at 6:23 PM Craig Russell  wrote:

> I'd like to ask on behalf of the DB PMC for the infra tool closer.lua to
> be enhanced a bit for use by projects.
>
> It will be much easier for us if we can simply link to e.g.
>
> https://www.apache.org/dyn/closer.lua/db/jdo/3.2/jdo-3.2-source-release.tar.gz
> and have that page contain the .asc and .sha512 as well.
>
> Even better if closer.lua can also find the relevant KEYS file and link to
> it when describing how to check the release.
>
> With this approach, it is easier for the project and less error-prone to
> maintain the download page(s). We can include the current release as well
> as archived releases on the same download page and omit the links to the
> .asc and .sha512 and KEYS files.
>
> Please let us know how we can help.
>
> Regards,
> Craig
>
>
> On Apr 16, 2022, at 2:01 AM, Greg Stein  wrote:
>
> On Thu, Apr 14, 2022 at 4:43 PM sebb  wrote:
>
>> On Thu, 14 Apr 2022 at 20:52, Christopher  wrote:
>> >
>> > Doing this would probably restrict INFRA's ability to adapt it to
>> > their changing needs, as projects would become dependent on it.
>>
>> Projects are already dependent on the page.
>>
>
> Correct. We *want* projects to use closer.lua. It provides a control point
> to direct users to the appropriate location to download. Today, it
> primarily sends them to our CDN and to an EU download location.
>
> Should those decisions ever change, we want projects to be using
> closer.lua to effect those changes.
>
>
>> I think it would make it easier for Infra to make changes, as they
>> would be able to adjust it.
>>
>
> Yes.
>
>
>> > Probably best to leave the mirror-management page independent from the
>> > project download pages, since they serve the needs of separate groups.
>>
>> I was not suggesting replacing these, for those projects that want to
>> customise the pages.
>>
>
> Today, we no longer have a mirror system. But requiring projects to use
> closer.lua ensures that we can swap in that future option.
>
> All that said, as I recall: closer.lua provides support for .ezt pages so
> that projects can provide custom download pages. Maybe there is a way to
> provide the needed variables to projects' custom download page templates.
>
> Or, they can just keep using their pages.
>
> The download pages are owned by the TLPs. Infra isn't gonna interfere with
> them. Should any TLPs want more features, then they can ask. We haven't
> seen any requests in years from the TLPs.
>
> Cheers,
> Greg
> InfraAdmin, ASF
>
>
>
> Craig L Russell
> c...@apache.org
>
>

Re: Extend closer.lua to act as a download page

[Craig Russell]

I'd like to ask on behalf of the DB PMC for the infra tool closer.lua to be 
enhanced a bit for use by projects.

It will be much easier for us if we can simply link to e.g.
https://www.apache.org/dyn/closer.lua/db/jdo/3.2/jdo-3.2-source-release.tar.gz
and have that page contain the .asc and .sha512 as well.

Even better if closer.lua can also find the relevant KEYS file and link to it 
when describing how to check the release.

With this approach, it is easier for the project and less error-prone to 
maintain the download page(s). We can include the current release as well as 
archived releases on the same download page and omit the links to the .asc and 
.sha512 and KEYS files.

Please let us know how we can help.

Regards,
Craig


> On Apr 16, 2022, at 2:01 AM, Greg Stein  wrote:
> 
> On Thu, Apr 14, 2022 at 4:43 PM sebb  > wrote:
> On Thu, 14 Apr 2022 at 20:52, Christopher  > wrote:
> >
> > Doing this would probably restrict INFRA's ability to adapt it to
> > their changing needs, as projects would become dependent on it.
> 
> Projects are already dependent on the page.
> 
> Correct. We *want* projects to use closer.lua. It provides a control point to 
> direct users to the appropriate location to download. Today, it primarily 
> sends them to our CDN and to an EU download location.
> 
> Should those decisions ever change, we want projects to be using closer.lua 
> to effect those changes.
>  
> I think it would make it easier for Infra to make changes, as they
> would be able to adjust it.
> 
> Yes.
>  
> > Probably best to leave the mirror-management page independent from the
> > project download pages, since they serve the needs of separate groups.
> 
> I was not suggesting replacing these, for those projects that want to
> customise the pages.
> 
> Today, we no longer have a mirror system. But requiring projects to use 
> closer.lua ensures that we can swap in that future option.
> 
> All that said, as I recall: closer.lua provides support for .ezt pages so 
> that projects can provide custom download pages. Maybe there is a way to 
> provide the needed variables to projects' custom download page templates.
> 
> Or, they can just keep using their pages.
> 
> The download pages are owned by the TLPs. Infra isn't gonna interfere with 
> them. Should any TLPs want more features, then they can ask. We haven't seen 
> any requests in years from the TLPs.
> 
> Cheers,
> Greg
> InfraAdmin, ASF
>  

Craig L Russell
c...@apache.org