Re: upgrade to openssl 3.0.0

[Ken Cunningham]

That is exactly the approach that I don't find workable, Chris.

Specifically:

1. every single port (think: rust, ghc, blah blah) will have to be needlessly 
managed to build against an alternate openssl location.
2. there will be no default openssl ever in prefix.

So I am not on board for trying to implement that myself, but should that be 
the selected course I will watch from the sidelines as others proceed down that 
path.

K

> On Oct 2, 2021, at 12:02 PM, Chris Jones  wrote:
> 
> Hi,
> 
> Personally I would strongly recommend following a migration path that does 
> not require an enmass change but allows ports to migrate  at their own pace. 
> I would personally very much recommend an approach similar to that we used 
> for boost recently
> 
> 1. Introduced a set of new isolated opensslXYZ ports that install specific 
> versions into isolated install areas under libexec.
> 2. Create a new portgroup that handles the confirmation of ports to pick the 
> openssl version they wish to use, with some default, and performs various 
> standard configurations for standard build systems such that for most ports 
> they work out the box with the new portgroup. For those that don’t provide 
> various proc methods that return the configured install paths such that ports 
> can used them to configure themselves as required.
> 3. Make the current openssl port a basic shim port that just installs sym 
> links into the primary install area to the default openssl version (for now 
> 1.1.1), so ports building against the current openssl port carry on working 
> just as they are now, but are redirected to use the same versioned 1.1.1 port 
> as those using the PG.
> 
> Once this is in place, ports can be migrated to use the new PG individually 
> as and when we want.
> 
> Chris
> 
>> On 2 Oct 2021, at 6:17 pm, Ken Cunningham  
>> wrote:
>> 
>> 
>> openssl 3.0.0 is out, with a new and very favourable Apache-2 license that 
>> will let many previously non-distributable ports become distributable.  
>> However, there are 758 ports that indicate they link against openssl. That 
>> is a daunting number of ports indeed.
>> 
>> One option might be to move all of our openssl ports (1.0, 1.1, and 3.x) 
>> into subdirectories out of ${prefix}, have no default openssl installed in 
>> $[prefix} directly, create a new PortGroup, openssl_select or some such, add 
>> that PG to all 758 ports, default it to openssl 1.1.1, and then allow people 
>> to gradually move the 758 ports to openssl 3.0.0 as they get to it.
>> 
>> Although that is possible with suitable effort, I don’t think that is 
>> workable for many reasons, the most obvious being that we would then have to 
>> modify every single port in some way to find an openssl installed into a 
>> non-default prefix. Creating the PG and adding it to 758 ports might be work 
>> enough, but then finding the right way to force all 758 ports to build 
>> properly against an openssl that is not in the default prefix is the real 
>> horror, and seems like a nightmare of wasted labours.
>> 
>> So it would appear that the same option that was chosen last time, ie 
>> upgrade the default openssl in ${prefix} to the newer openssl, and then fix 
>> the subset of ports that will not build against it to use an older openssl 
>> appears both the better option and lot less work (assuming most ports do 
>> build against openssl 3.0.0, which seems to be the case so far). Some will 
>> disagree, but I put it to you that it is going to be far less work in the 
>> end to force a few % of the ports to a specific alternate openssl than force 
>> all of them, all the time, forever.
>> 
>> Most things I have attempted to rebuild over the past few days have rebuilt 
>> without any issues, but a few things don’t build with openssl 3.0.0 yet and 
>> will need to stay with openssl 1.1.1 for a while until patched or updated 
>> (or forever). That will require both forcing those ports to find an 
>> alternate openssl installation, and also (the tricky part) forcing them to 
>> ignore the openssl in the default prefix.
>> 
>> To support this, there is a new openssl11 port that provides openssl 1.1.1 
>> tucked away in a subdir, much like we have openssl10, and a few new options 
>> were added to the old_openssl PortGroup to allow most ports to be forced to 
>> the alternate openssl with minimal fuss. Add the PortGroup, spec the branch, 
>> and choose the method, for the most part.
>> 
>> If this plan holds, I would anticipate that we move ports that we find need 
>> to stay on openssl 1.1.1 to openssl11 using the old_openssl PortGroup soon 
>> or now, before we upgrade to openssl 3.0.0 to minimize fuss. Then once we 
>> have done that, upgrading the default openssl to 3.0.0 and revbumping the 
>> deps would occur.
>> 
>> So I am asking that anyone with an interest in a port that uses openssl 
>> please try building it against openssl 3.0.0 in the PR below, and if there 
>> is a problem that can’t be solved by 

Re: upgrade to openssl 3.0.0

[Chris Jones]

Hi,

Personally I would strongly recommend following a migration path that does not 
require an enmass change but allows ports to migrate  at their own pace. I 
would personally very much recommend an approach similar to that we used for 
boost recently

1. Introduced a set of new isolated opensslXYZ ports that install specific 
versions into isolated install areas under libexec.
2. Create a new portgroup that handles the confirmation of ports to pick the 
openssl version they wish to use, with some default, and performs various 
standard configurations for standard build systems such that for most ports 
they work out the box with the new portgroup. For those that don’t provide 
various proc methods that return the configured install paths such that ports 
can used them to configure themselves as required.
3. Make the current openssl port a basic shim port that just installs sym links 
into the primary install area to the default openssl version (for now 1.1.1), 
so ports building against the current openssl port carry on working just as 
they are now, but are redirected to use the same versioned 1.1.1 port as those 
using the PG.

Once this is in place, ports can be migrated to use the new PG individually as 
and when we want.

Chris

> On 2 Oct 2021, at 6:17 pm, Ken Cunningham  
> wrote:
> 
> 
> openssl 3.0.0 is out, with a new and very favourable Apache-2 license that 
> will let many previously non-distributable ports become distributable.  
> However, there are 758 ports that indicate they link against openssl. That is 
> a daunting number of ports indeed.
> 
> One option might be to move all of our openssl ports (1.0, 1.1, and 3.x) into 
> subdirectories out of ${prefix}, have no default openssl installed in 
> $[prefix} directly, create a new PortGroup, openssl_select or some such, add 
> that PG to all 758 ports, default it to openssl 1.1.1, and then allow people 
> to gradually move the 758 ports to openssl 3.0.0 as they get to it.
> 
> Although that is possible with suitable effort, I don’t think that is 
> workable for many reasons, the most obvious being that we would then have to 
> modify every single port in some way to find an openssl installed into a 
> non-default prefix. Creating the PG and adding it to 758 ports might be work 
> enough, but then finding the right way to force all 758 ports to build 
> properly against an openssl that is not in the default prefix is the real 
> horror, and seems like a nightmare of wasted labours.
> 
> So it would appear that the same option that was chosen last time, ie upgrade 
> the default openssl in ${prefix} to the newer openssl, and then fix the 
> subset of ports that will not build against it to use an older openssl 
> appears both the better option and lot less work (assuming most ports do 
> build against openssl 3.0.0, which seems to be the case so far). Some will 
> disagree, but I put it to you that it is going to be far less work in the end 
> to force a few % of the ports to a specific alternate openssl than force all 
> of them, all the time, forever.
> 
> Most things I have attempted to rebuild over the past few days have rebuilt 
> without any issues, but a few things don’t build with openssl 3.0.0 yet and 
> will need to stay with openssl 1.1.1 for a while until patched or updated (or 
> forever). That will require both forcing those ports to find an alternate 
> openssl installation, and also (the tricky part) forcing them to ignore the 
> openssl in the default prefix.
> 
> To support this, there is a new openssl11 port that provides openssl 1.1.1 
> tucked away in a subdir, much like we have openssl10, and a few new options 
> were added to the old_openssl PortGroup to allow most ports to be forced to 
> the alternate openssl with minimal fuss. Add the PortGroup, spec the branch, 
> and choose the method, for the most part.
> 
> If this plan holds, I would anticipate that we move ports that we find need 
> to stay on openssl 1.1.1 to openssl11 using the old_openssl PortGroup soon or 
> now, before we upgrade to openssl 3.0.0 to minimize fuss. Then once we have 
> done that, upgrading the default openssl to 3.0.0 and revbumping the deps 
> would occur.
> 
> So I am asking that anyone with an interest in a port that uses openssl 
> please try building it against openssl 3.0.0 in the PR below, and if there is 
> a problem that can’t be solved by an update or a patch, consider trying to 
> use the old_openssl PortGroup to fix the build and move it over. As there are 
> so many ports, it would help if people pitched in with the ones that are 
> important to them.
> 
> The openssl 3.0.0 upgrade PR is here:
> 
> https://github.com/macports/macports-ports/pull/12410
> 
> and my own WIP repo of a few ports that I have found won’t build with openssl 
> 3.0.0 at present and fixed to use the old_openssl PG is here:
> 
> https://github.com/kencu/mybigsuroverlay/
> 
> the idea is that the ports in that repo would be moved into the main repo 

Re: upgrade to openssl 3.0.0

[Ken Cunningham]

All the pythons build against openssl 3.0.0, so that python issue with all it's 
trail-down conflicts will disappear with the upgrade and python revbump.

A very very large % of ports do as well (and those that don't now soon will, as 
everyone moves to openssl 3.0.0 as the default, which homebrew did pretty much 
the second openssl 3 came out).

So I am hoping that this upgrade is the beginning of the end of this particular 
PITA for all of us.

Ken


> On Oct 2, 2021, at 11:37 AM, Jason Liu  wrote:
> 
> That was also the Blender devs' claim, which I assume is why they decided it 
> wasn't necessary to include the GPL-OpenSSL exception text, since any 
> licensing conflicts would self-resolve once Blender starts using OpenSSL 3.0. 
> But currently, their pre-built release binary downloads and compiles OpenSSL 
> 1.1.1g, which is still under the OpenSSL license, not Apache 2.0.
> 
> -- 
> Jason Liu
> 
> 
> On Sat, Oct 2, 2021 at 2:25 PM Joshua Root  > wrote:
> Blender is GPL-2+, which means it can be distributed when linked with 
> OpenSSL 3.0, since GPL-3 is compatible with Apache-2.
> 
> - Josh
> 
> On 2021-10-3 05:09 , Jason Liu wrote:
> > I hope the question that I'm about to ask doesn't induce 
> > "Inception"-style migraines, but since it directly relates to one of the 
> > ports I maintain, here goes. What if one of my ports indirectly uses 
> > openssl through one of its dependencies, and the licensing terms in the 
> > current version of my port only covers openssl 1.1.1, but not 3.0?
> > 
> > To use a specific example, Blender uses openssl 1.1.1 indirectly, by way 
> > of using networking through python. I contacted the Blender devs about 
> > this, and even though they acknowledged that they were unknowingly using 
> > OpenSSL without including the OpenSSL license, the only thing they ended 
> > up doing was to add the OpenSSL license to their licenses directory. 
> > They refuse, even now, to add the chunk of text specifying the 
> > GPL-OpenSSL exception to their licenses. Somehow their legal team seems 
> > to think that's enough to allow them to distribute pre-compiled 
> > binaries, but the MacPorts automatic license checking is flagging my 
> > Blender port as being non-distributable. Since my port is a couple of 
> > versions behind the latest release of Blender (there are some new 
> > dependent libraries that I need to submit first), how should we specify 
> > in our Portfiles that one of the dependencies should continue using the 
> > old openssl11, without adding the old_openssl PortGroup, and thus a 
> > direct dependency on openssl? Does this mean that the dependencies which 
> > are directly dependent on openssl will need new variants, e.g. +openssl 
> > and +openssl11?
> > 
> > -- 
> > Jason Liu

Re: upgrade to openssl 3.0.0

[Jason Liu]

That was also the Blender devs' claim, which I assume is why they decided
it wasn't necessary to include the GPL-OpenSSL exception text, since any
licensing conflicts would self-resolve once Blender starts using OpenSSL
3.0. But currently, their pre-built release binary downloads and compiles
OpenSSL 1.1.1g, which is still under the OpenSSL license, not Apache 2.0.

-- 
Jason Liu


On Sat, Oct 2, 2021 at 2:25 PM Joshua Root  wrote:

> Blender is GPL-2+, which means it can be distributed when linked with
> OpenSSL 3.0, since GPL-3 is compatible with Apache-2.
>
> - Josh
>
> On 2021-10-3 05:09 , Jason Liu wrote:
> > I hope the question that I'm about to ask doesn't induce
> > "Inception"-style migraines, but since it directly relates to one of the
> > ports I maintain, here goes. What if one of my ports indirectly uses
> > openssl through one of its dependencies, and the licensing terms in the
> > current version of my port only covers openssl 1.1.1, but not 3.0?
> >
> > To use a specific example, Blender uses openssl 1.1.1 indirectly, by way
> > of using networking through python. I contacted the Blender devs about
> > this, and even though they acknowledged that they were unknowingly using
> > OpenSSL without including the OpenSSL license, the only thing they ended
> > up doing was to add the OpenSSL license to their licenses directory.
> > They refuse, even now, to add the chunk of text specifying the
> > GPL-OpenSSL exception to their licenses. Somehow their legal team seems
> > to think that's enough to allow them to distribute pre-compiled
> > binaries, but the MacPorts automatic license checking is flagging my
> > Blender port as being non-distributable. Since my port is a couple of
> > versions behind the latest release of Blender (there are some new
> > dependent libraries that I need to submit first), how should we specify
> > in our Portfiles that one of the dependencies should continue using the
> > old openssl11, without adding the old_openssl PortGroup, and thus a
> > direct dependency on openssl? Does this mean that the dependencies which
> > are directly dependent on openssl will need new variants, e.g. +openssl
> > and +openssl11?
> >
> > --
> > Jason Liu
>

Re: upgrade to openssl 3.0.0

[Joshua Root]

Blender is GPL-2+, which means it can be distributed when linked with 
OpenSSL 3.0, since GPL-3 is compatible with Apache-2.


- Josh

On 2021-10-3 05:09 , Jason Liu wrote:
I hope the question that I'm about to ask doesn't induce 
"Inception"-style migraines, but since it directly relates to one of the 
ports I maintain, here goes. What if one of my ports indirectly uses 
openssl through one of its dependencies, and the licensing terms in the 
current version of my port only covers openssl 1.1.1, but not 3.0?


To use a specific example, Blender uses openssl 1.1.1 indirectly, by way 
of using networking through python. I contacted the Blender devs about 
this, and even though they acknowledged that they were unknowingly using 
OpenSSL without including the OpenSSL license, the only thing they ended 
up doing was to add the OpenSSL license to their licenses directory. 
They refuse, even now, to add the chunk of text specifying the 
GPL-OpenSSL exception to their licenses. Somehow their legal team seems 
to think that's enough to allow them to distribute pre-compiled 
binaries, but the MacPorts automatic license checking is flagging my 
Blender port as being non-distributable. Since my port is a couple of 
versions behind the latest release of Blender (there are some new 
dependent libraries that I need to submit first), how should we specify 
in our Portfiles that one of the dependencies should continue using the 
old openssl11, without adding the old_openssl PortGroup, and thus a 
direct dependency on openssl? Does this mean that the dependencies which 
are directly dependent on openssl will need new variants, e.g. +openssl 
and +openssl11?


--
Jason Liu

Re: upgrade to openssl 3.0.0

[Jason Liu]

I hope the question that I'm about to ask doesn't induce "Inception"-style
migraines, but since it directly relates to one of the ports I maintain,
here goes. What if one of my ports indirectly uses openssl through one
of its dependencies, and the licensing terms in the current version of my
port only covers openssl 1.1.1, but not 3.0?

To use a specific example, Blender uses openssl 1.1.1 indirectly, by way
of using networking through python. I contacted the Blender devs about
this, and even though they acknowledged that they were unknowingly using
OpenSSL without including the OpenSSL license, the only thing they ended up
doing was to add the OpenSSL license to their licenses directory. They
refuse, even now, to add the chunk of text specifying the GPL-OpenSSL
exception to their licenses. Somehow their legal team seems to think that's
enough to allow them to distribute pre-compiled binaries, but the MacPorts
automatic license checking is flagging my Blender port as being
non-distributable. Since my port is a couple of versions behind the latest
release of Blender (there are some new dependent libraries that I need to
submit first), how should we specify in our Portfiles that one of the
dependencies should continue using the old openssl11, without adding the
old_openssl PortGroup, and thus a direct dependency on openssl? Does this
mean that the dependencies which are directly dependent on openssl will
need new variants, e.g. +openssl and +openssl11?

-- 
Jason Liu


On Sat, Oct 2, 2021 at 1:17 PM Ken Cunningham <
ken.cunningham.web...@gmail.com> wrote:

> openssl 3.0.0 is out, with a new and very favourable Apache-2 license that
> will let many previously non-distributable ports become distributable.
> However, there are 758 ports that indicate they link against openssl. That
> is a daunting number of ports indeed.
>
> One option might be to move all of our openssl ports (1.0, 1.1, and 3.x)
> into subdirectories out of ${prefix}, have no default openssl installed in
> $[prefix} directly, create a new PortGroup, openssl_select or some such,
> add that PG to all 758 ports, default it to openssl 1.1.1, and then allow
> people to gradually move the 758 ports to openssl 3.0.0 as they get to it.
>
> Although that is possible with suitable effort, I don’t think that is
> workable for many reasons, the most obvious being that we would then have
> to modify every single port in some way to find an openssl installed into a
> non-default prefix. Creating the PG and adding it to 758 ports might be
> work enough, but then finding the right way to force all 758 ports to build
> properly against an openssl that is not in the default prefix is the real
> horror, and seems like a nightmare of wasted labours.
>
> So it would appear that the same option that was chosen last time, ie
> upgrade the default openssl in ${prefix} to the newer openssl, and then fix
> the subset of ports that will not build against it to use an older openssl
> appears both the better option and lot less work (assuming most ports do
> build against openssl 3.0.0, which seems to be the case so far). Some will
> disagree, but I put it to you that it is going to be far less work in the
> end to force a few % of the ports to a specific alternate openssl than
> force all of them, all the time, forever.
>
> Most things I have attempted to rebuild over the past few days have
> rebuilt without any issues, but a few things don’t build with openssl 3.0.0
> yet and will need to stay with openssl 1.1.1 for a while until patched or
> updated (or forever). That will require both forcing those ports to find an
> alternate openssl installation, and also (the tricky part) forcing them to
> ignore the openssl in the default prefix.
>
> To support this, there is a new openssl11 port that provides openssl 1.1.1
> tucked away in a subdir, much like we have openssl10, and a few new options
> were added to the old_openssl PortGroup to allow most ports to be forced to
> the alternate openssl with minimal fuss. Add the PortGroup, spec the
> branch, and choose the method, for the most part.
>
> If this plan holds, I would anticipate that we move ports that we find
> need to stay on openssl 1.1.1 to openssl11 using the old_openssl PortGroup
> soon or now, before we upgrade to openssl 3.0.0 to minimize fuss. Then once
> we have done that, upgrading the default openssl to 3.0.0 and revbumping
> the deps would occur.
>
> So I am asking that anyone with an interest in a port that uses openssl
> please try building it against openssl 3.0.0 in the PR below, and if there
> is a problem that can’t be solved by an update or a patch, consider trying
> to use the old_openssl PortGroup to fix the build and move it over. As
> there are so many ports, it would help if people pitched in with the ones
> that are important to them.
>
> The openssl 3.0.0 upgrade PR is here:
>
> https://github.com/macports/macports-ports/pull/12410
>
> and my own WIP repo of a few ports that I have found 

upgrade to openssl 3.0.0

[Ken Cunningham]

openssl 3.0.0 is out, with a new and very favourable Apache-2 license that will 
let many previously non-distributable ports become distributable.  However, 
there are 758 ports that indicate they link against openssl. That is a daunting 
number of ports indeed.

One option might be to move all of our openssl ports (1.0, 1.1, and 3.x) into 
subdirectories out of ${prefix}, have no default openssl installed in $[prefix} 
directly, create a new PortGroup, openssl_select or some such, add that PG to 
all 758 ports, default it to openssl 1.1.1, and then allow people to gradually 
move the 758 ports to openssl 3.0.0 as they get to it.

Although that is possible with suitable effort, I don’t think that is workable 
for many reasons, the most obvious being that we would then have to modify 
every single port in some way to find an openssl installed into a non-default 
prefix. Creating the PG and adding it to 758 ports might be work enough, but 
then finding the right way to force all 758 ports to build properly against an 
openssl that is not in the default prefix is the real horror, and seems like a 
nightmare of wasted labours.

So it would appear that the same option that was chosen last time, ie upgrade 
the default openssl in ${prefix} to the newer openssl, and then fix the subset 
of ports that will not build against it to use an older openssl appears both 
the better option and lot less work (assuming most ports do build against 
openssl 3.0.0, which seems to be the case so far). Some will disagree, but I 
put it to you that it is going to be far less work in the end to force a few % 
of the ports to a specific alternate openssl than force all of them, all the 
time, forever.

Most things I have attempted to rebuild over the past few days have rebuilt 
without any issues, but a few things don’t build with openssl 3.0.0 yet and 
will need to stay with openssl 1.1.1 for a while until patched or updated (or 
forever). That will require both forcing those ports to find an alternate 
openssl installation, and also (the tricky part) forcing them to ignore the 
openssl in the default prefix.

To support this, there is a new openssl11 port that provides openssl 1.1.1 
tucked away in a subdir, much like we have openssl10, and a few new options 
were added to the old_openssl PortGroup to allow most ports to be forced to the 
alternate openssl with minimal fuss. Add the PortGroup, spec the branch, and 
choose the method, for the most part.

If this plan holds, I would anticipate that we move ports that we find need to 
stay on openssl 1.1.1 to openssl11 using the old_openssl PortGroup soon or now, 
before we upgrade to openssl 3.0.0 to minimize fuss. Then once we have done 
that, upgrading the default openssl to 3.0.0 and revbumping the deps would 
occur.

So I am asking that anyone with an interest in a port that uses openssl please 
try building it against openssl 3.0.0 in the PR below, and if there is a 
problem that can’t be solved by an update or a patch, consider trying to use 
the old_openssl PortGroup to fix the build and move it over. As there are so 
many ports, it would help if people pitched in with the ones that are important 
to them.

The openssl 3.0.0 upgrade PR is here:

https://github.com/macports/macports-ports/pull/12410 


and my own WIP repo of a few ports that I have found won’t build with openssl 
3.0.0 at present and fixed to use the old_openssl PG is here:

https://github.com/kencu/mybigsuroverlay/ 


the idea is that the ports in that repo would be moved into the main repo as we 
work through the main ones that use openssl at least.


K

PS. One thing I have not yet sorted out how to do is how to support libressl in 
the setting of needing to force a port to openssl 1.1.1 using the old_openssl 
PortGroup. Open to any thoughts on this. This case wsa basically just ignored 
last time with openssl10, and we may have to do the same again unless someone 
has a bright idea.

RE: kibana submission node version conflict issue

[Christopher Nielsen]

> On Oct 2 2021, at 05:30, Ryan Schmidt wrote:
> 
>> On Sep 2, 2021, at 05:12, Steven Smith wrote:
>> 
>> How can this node version conflict be resolved?
> 
> The node ports should be fixed so that they do not conflict with one another.

Agreed, and tracked by open issue:

https://trac.macports.org/ticket/63035

Re: kibana submission node version conflict issue

[Ryan Schmidt]

On Sep 2, 2021, at 05:12, Steven Smith wrote:

> How can this node version conflict be resolved?

The node ports should be fixed so that they do not conflict with one another.