Re: Enterprise proxies and mirrors issues when updating

[Jaroslav Tulach]

Dne neděle 6. září 2020 18:03:49 CEST, Neil C Smith napsal(a):
> On Sun, 6 Sep 2020 at 15:58, Matthias Bläsing  
wrote:
> > > Of course, modify your IDE to use the URL
> > > to the
> > > selected mirror instead of the default randomly distributing URL.
> > 
> > this is not a good idea, unless you fully trust the mirror operator.
> > The updates.xml.gz _must_ be downloaded from apache infrastructure via
> > a https connection.

I see.  The mirror would have to be trusted.

> Incidentally, this is also not a good idea because you'll never
> receive any updates at all!  The updates.xml on the mirrors is fixed
> at release.  The one we serve to the IDE has the relevant updated
> modules spliced in.

I see. One would have change this URL from time to time manually.

Looks like I solved the problem with redirection to different mirrors, but 
caused two other issues. "Fix a bug, introduce two" ;-)

-jt




-
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

Re: Enterprise proxies and mirrors issues when updating

[Neil C Smith]

On Sun, 6 Sep 2020 at 15:58, Matthias Bläsing  wrote:
> > Of course, modify your IDE to use the URL
> > to the
> > selected mirror instead of the default randomly distributing URL.
> >
> this is not a good idea, unless you fully trust the mirror operator.
> The updates.xml.gz _must_ be downloaded from apache infrastructure via
> a https connection.

Incidentally, this is also not a good idea because you'll never
receive any updates at all!  The updates.xml on the mirrors is fixed
at release.  The one we serve to the IDE has the relevant updated
modules spliced in.

Best wishes,

Neil

-
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

Re: Enterprise proxies and mirrors issues when updating

[Neil C Smith]

On Sun, 6 Sep 2020, 15:57 Matthias Bläsing, 
wrote:

> > The IDE treats as relative from the original link, not the end point.
> > This is the cause of problems, hence question about absolute linked
> > catalogs per mirror.
>
> Even the endpoint, that serves the updates.xml.gz is not the right
> point for the binary downloads.
>

Well, no, but thinking of older problem with the netbeans.a.o to vm
redirect, where you get rejected if you hit it too much (eg. cluster
download). Links relative to endpoint of updates.xml would still redirect
to mirrors, but fix that problem. Absolute links a better option for other
issues as well though.

Best wishes,

Neil

>

Re: Enterprise proxies and mirrors issues when updating

[Matthias Bläsing]

Hi,

Am Sonntag, den 06.09.2020, 05:50 +0200 schrieb Jaroslav Tulach:
> Hello Jean-Marc Borer.
> 
> > The problem is that  https://netbeans.apache.org/nb/updates/12.0/
> > 
> > redirects to
> > mirrors that are not white listed. The list changes too often to be
> > maintained by me and accepted by my company.
> 
> When I click the above URL I get to:
> https://apache.miloslavbrada.cz/netbeans/netbeans/12.0/nbms/updates.xml.gz
> I assume all the links from `updates.xml` are then relative. 
> 
> Why don't you white list `apache.miloslavbrada.cz` or any other
> Apache mirror 
> that you are redirected to? Of course, modify your IDE to use the URL
> to the 
> selected mirror instead of the default randomly distributing URL.
> 

this is not a good idea, unless you fully trust the mirror operator.
The updates.xml.gz _must_ be downloaded from apache infrastructure via
a https connection.

The updates.xml.gz acts as a trust anchor and thus it needs to come
from a trustworthy source. The mirror network is not controlled by
apache and thus every operator could inject malicous data. We protect
against this:

- The updates.xml.gz comes from trusted apache infrastructure
- the updates.xml.gz holds cryptographic hashes of the artifacts, 
  artifacts whose hashes don't match the updates.xml.gz value after
  download are rejected.

This allows downloads to happen from the mirrors an still ensuring,
that they are identical to the download from the main apache mirror.
The only requirement is that the updates.xml.gz must come from a
trusted source.

Matthias


-
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

Re: Enterprise proxies and mirrors issues when updating

[Matthias Bläsing]

Am Sonntag, den 06.09.2020, 08:57 +0100 schrieb Neil C Smith:
> On Sun, 6 Sep 2020 at 04:50, Jaroslav Tulach  
> wrote:
> > When I click the above URL I get to:
> > https://apache.miloslavbrada.cz/netbeans/netbeans/12.0/nbms/updates.xml.gz
> 
> Are you sure?!  I can't check at the moment, can't seem to connect to
> a lot of Apache right now for some reason.  Trying to check with -
> 
> curl -sLD - https://netbeans.apache.org/nb/updates/12.0/updates.xml.gz
> -o /dev/null
> 
> If you're getting updatesxml.gz from a mirror we have an
> infrastructure problem that needs sorting out!  The catalog should
> always be served from the NetBeans VM, only the nbms from the mirrors
> - catalog on the mirrors is not correct.

"The above URL" is 

https://netbeans.apache.org/nb/updates/12.0/ 

and that is correctly redirected to one of the mirrors, but this:

https://netbeans.apache.org/nb/updates/12.0/updates.xml.gz

is not. This is the intended behaviour. If a user manually downloads
the file from a mirror, he is required to verify the signaure by ASF
policy. If that happens - great, if not, the user shot himself into his
foot.


> > I assume all the links from `updates.xml` are then relative.
> 
> The IDE treats as relative from the original link, not the end point.
> This is the cause of problems, hence question about absolute linked
> catalogs per mirror.

Even the endpoint, that serves the updates.xml.gz is not the right
point for the binary downloads. See my other reply for a possible
solution, handling the creation of absolute links on the apache
infrastructure.

Greetings

Matthias


-
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

Re: Enterprise proxies and mirrors issues when updating

[Neil C Smith]

On Sun, 6 Sep 2020 at 04:50, Jaroslav Tulach  wrote:
> When I click the above URL I get to:
> https://apache.miloslavbrada.cz/netbeans/netbeans/12.0/nbms/updates.xml.gz

Are you sure?!  I can't check at the moment, can't seem to connect to
a lot of Apache right now for some reason.  Trying to check with -

curl -sLD - https://netbeans.apache.org/nb/updates/12.0/updates.xml.gz
-o /dev/null

If you're getting updatesxml.gz from a mirror we have an
infrastructure problem that needs sorting out!  The catalog should
always be served from the NetBeans VM, only the nbms from the mirrors
- catalog on the mirrors is not correct.

> I assume all the links from `updates.xml` are then relative.

The IDE treats as relative from the original link, not the end point.
This is the cause of problems, hence question about absolute linked
catalogs per mirror.

Best wishes,

Neil

-
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

Re: Enterprise proxies and mirrors issues when updating

[Jaroslav Tulach]

Hello Jean-Marc Borer.

> The problem is that  https://netbeans.apache.org/nb/updates/12.0/
>  redirects to
> mirrors that are not white listed. The list changes too often to be
> maintained by me and accepted by my company.

When I click the above URL I get to:
https://apache.miloslavbrada.cz/netbeans/netbeans/12.0/nbms/updates.xml.gz
I assume all the links from `updates.xml` are then relative. 

Why don't you white list `apache.miloslavbrada.cz` or any other Apache mirror 
that you are redirected to? Of course, modify your IDE to use the URL to the 
selected mirror instead of the default randomly distributing URL.

-jt




-
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

Re: Enterprise proxies and mirrors issues when updating

[Matthias Bläsing]

Hi Neil,

Am Samstag, den 05.09.2020, 13:52 +0100 schrieb Neil C Smith:
> On Sat, 5 Sep 2020 at 13:02, Matthias Bläsing  
> wrote:
> > I demonstrated in the past, that we can deliver update.xml with a
> > single mirror selected:
> > 
> > https://github.com/matthiasblaesing/netbeans-tools/tree/proxy-chooser
> > 
> > If there is interest, I'm willing to put more work into it, but I won't
> > do it just because.
> 
> Definitely yes if it can support automatically picking a mirror too?
> Can't remember if that's the case when discussed before.  Always
> having NBMs linked via an absolute link so everything is downloaded
> from one mirror and the VM is never hit for all the redirects would be
> a big plus in my opinion.

Here you are:

https://github.com/matthiasblaesing/netbeans-tools/commits/proxy-chooser

We need to see if this needs further optimization, but it should do the
trick. There is a README.md, that might be of interest:

https://github.com/matthiasblaesing/netbeans-tools/blob/7224db4ff5f4edbaf53a9b2f2250f0068060abd8/proxy-chooser/README.md

That file shows the options, that the skript gives. The sample hit my
server, where the branch is deployed.

Greetings

Matthias





-
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

Re: Enterprise proxies and mirrors issues when updating

[Neil C Smith]

On Sat, 5 Sep 2020 at 13:02, Matthias Bläsing  wrote:
> I demonstrated in the past, that we can deliver update.xml with a
> single mirror selected:
>
> https://github.com/matthiasblaesing/netbeans-tools/tree/proxy-chooser
>
> If there is interest, I'm willing to put more work into it, but I won't
> do it just because.

Definitely yes if it can support automatically picking a mirror too?
Can't remember if that's the case when discussed before.  Always
having NBMs linked via an absolute link so everything is downloaded
from one mirror and the VM is never hit for all the redirects would be
a big plus in my opinion.

Best wishes,

Neil

-
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

Re: Enterprise proxies and mirrors issues when updating

[Matthias Bläsing]

Hi,

Am Donnerstag, den 03.09.2020, 11:42 + schrieb Jean-Marc Borer:
> Wouldn't it be possible to have an update center that is not relying on
> mirrors so that I can have a single place from which my company allows me
> to download Java items from?

this turns your/your companies problem into a problem of the ASF. The
mirror networks is a donation to the ASF to keep traffic away from the
central download server. So no, using the central server is not a
solution.

I demonstrated in the past, that we can deliver update.xml with a
single mirror selected:

https://github.com/matthiasblaesing/netbeans-tools/tree/proxy-chooser

If there is interest, I'm willing to put more work into it, but I won't
do it just because.

Matthias




-
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

Re: Enterprise proxies and mirrors issues when updating

[Neil C Smith]

On Thu, 3 Sep 2020 at 12:55, Jean-Marc Borer  wrote:
> The problem is that  https://netbeans.apache.org/nb/updates/12.0/
>  redirects to
> mirrors that are not white listed. The list changes too often to be
> maintained by me and accepted by my company.
>
> Is there a list somewhere of those mirrors?

At https://www.apache.org/mirrors/

> Ideally I would like to add a new update center that always points to the
> same domain. netbeans.org and apache.org are already white listed at our
> company.
>
> Any ideas?

If you're really stuck, you could try downloading manually from -

https://downloads.apache.org/netbeans/netbeans/12.0-u1/nbms/

You'll have to go through each cluster.

Or add the updates.xml there as a temporary update centre should work
I think - might need to disable the existing one?

I have to use that link for the RCP harness in my build scripts - the
Ant platform build scripts don't play well with mirrors either.

Note the text at https://downloads.apache.org/ - "Please do not
download releases from downloads.apache.org unless you absolutely have
to!"  But it is there as a fallback when no other choice is available.

Best wishes,

Neil

-
To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org
For additional commands, e-mail: dev-h...@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists

Re: Enterprise proxies and mirrors issues when updating

[Jean-Marc Borer]

The problem is that  https://netbeans.apache.org/nb/updates/12.0/
 redirects to
mirrors that are not white listed. The list changes too often to be
maintained by me and accepted by my company.

Is there a list somewhere of those mirrors?

Ideally I would like to add a new update center that always points to the
same domain. netbeans.org and apache.org are already white listed at our
company.

Any ideas?

Cheers,

JM

On Thu, Sep 3, 2020 at 11:42 AM Jean-Marc Borer  wrote:

> Hi guys,
> Same story again and again for me: I cannot update Netbeans through the
> update center because it redirects to random mirrors. At our company, they
> filter out everything related to java except for white listed sites. The
> process is long, tedious for me and not always successful to add new sites
> to the list.
>
> Wouldn't it be possible to have an update center that is not relying on
> mirrors so that I can have a single place from which my company allows me
> to download Java items from?
>
> Cheers,
>
> JM
>

Enterprise proxies and mirrors issues when updating

[Jean-Marc Borer]

Hi guys,
Same story again and again for me: I cannot update Netbeans through the
update center because it redirects to random mirrors. At our company, they
filter out everything related to java except for white listed sites. The
process is long, tedious for me and not always successful to add new sites
to the list.

Wouldn't it be possible to have an update center that is not relying on
mirrors so that I can have a single place from which my company allows me
to download Java items from?

Cheers,

JM