Re: Package managers and package versioning

[Jason Liu]

>
> It's my understanding, for example, that Homebrew allows the user to
> request to install the git master of a particular Homebrew formula, in
> effect allowing the user to (try to) install a version of the software that
> was never previously attempted by the formula's creator. This is what I was
> afraid was part of what was being proposed, and something I would not
> support adding to MacPorts.
>

I'm not that familiar with Homebrew, so this is the first time I'm hearing
about this sort of capability. None of the other PMSes that I have
experience with can do this. The idea of being able to (try to) install a
version of a package that doesn't even exist in a repository's metadata
database sounds insane to me.

If we continue the liberal approach with dependencies, then we begin from
> the assumption that a port can use any version of a dependency. Let's say
> libpng declares it can use any version of zlib. Then years later zlib 2 is
> released and it is incompatible, breaking all versions of libpng we've
> published. There's now a lot of work to go edit each previous libpng to
> indicate that it requires zlib < 2.
>

#!/bin/bash
for f in libpng/Portfile-*; do
sed -i -e s/(port:zlib.*)$/\1 < 2/ $f
done

I'm probably missing some backslashes, but I think you probably get the
idea. That doesn't seem like that much work to me.

Alternatively, suppose we use the conservative approach and declare that
> libpng requires zlib < 2. Or zlib < 1.3. Or zlib < 1.2.12. We have no way
> to know in what future version of zlib, if any, incompatibility might
> arise. Suppose we declare compatibility with zlib < 2, and years later zlib
> 2 comes along and maintains perfect backward compatibility with zlib 1.
> Then our libpng would continue to use the old zlib 1, even though it would
> work with zlib 2, thus making users use outdated dependencies unnecessarily
> and possibly exposing them to bugs that have already been fixed in newer
> versions. And the solution then is to do a lot of work to go edit each
> previous libpng to update its zlib compatibility list.
>

A sed statement wrapped inside a for loop would probably also help in this
case.

Either way, it's signing all port maintainers up for a lot of work. (Many
> port maintainers already don't do the currently expected amount of work to
> maintain their ports.) And all for the benefit of other ports being able to
> declare that they require an old version of libpng. We did have this
> problem once before, when some old ports were not compatible with libpng
> 1.4 when it came out. Rather than introducing a libpng12 compatibility
> port, we did the right thing and fixed other software to be compatible with
> libpng 1.4. If we had used a libpng12 port instead, we likely would never
> have noticed if and when those ports gained libpng 1.4 compatibility and
> would be forcing libpng12 on users unnecessarily for years to come.
>

However, I feel like this would be a good example of why having multiple
versions of a package's portfile available could be beneficial. Let's say
libpng 1.4 came out and it broke a bunch of old ports. If previous versions
of the portfile were available, like what I showed in my example of the
Blender Debian package, then it would be very easy to simply remove the
libpng 1.4 portfile from the repository, and all of the ports that were
broken would suddenly be fine again, because they would just go back to
using whichever portfile for libpng was already there. No need to increase
the epoch inside the portfile to try to back-rev to an older version of
libpng, no need to create a libpng12 compatibility port. And the port
maintainers would be able to go right back to working on fixing ports
locally on their test machines, without having to worry that the actual
global ports tree had been broken for users. (By the way, this is why most
flavors of Linux have multiple release channels, typically "stable",
"testing", and "unstable". Each release channel has its own PMS repository,
to avoid the situation where the global repository that users use gets
polluted due to a package incompatibility.)

And all for the benefit of other ports being able to declare that they
> require an old version of libpng.
>

If there's really such little benefit, then why are we even having such a
discussion? And why are there other PMSes which have this ability? Because
they enjoy torturing their package maintainers with lots of busy work?
Somehow I doubt it They wouldn't have gone to all the trouble of adding
such a feature unless it was able to reduce the amount of work they had to
do in some way.

This is one place where MacPort's activate/deactivate would really shine:
>> If someone wanted to install an old package, MacPorts would first
>> deactivate all of the dependencies that were too new, and then install the
>> old versions of the dependencies that are compatible with the old package.
>>
>
> If all versions of a port install to the same location on disk 

Re: Package managers and package versioning

[Ryan Schmidt]

On Oct 4, 2020, at 14:09, Jason Liu wrote:

>> Where by "this" you again mean the ability to specify a previously-available 
>> version, not an arbitrary version that was never in a portfile.
> 
> Yes, of course. If you look at the folder for the Blender package on the 
> Debian repo
> 
> http://deb.debian.org/debian/pool/main/b/blender/
> 
> from my previous example, there are many versions of Blender that were 
> skipped. For instance, they skipped version 2.82 and went from 2.79b right to 
> 2.82a. And they skipped versions 2.83.0-2.83.4, and went from 2.82a right to 
> 2.83.5. Obviously, there would be no way for a user to try to install Blender 
> version 2.83.2 using apt-get, because that version was never packaged (i.e. 
> spec files don't exist) for that particular version of Blender.

Let's try not to say "of course". Nothing is obvious in this discussion. Let us 
be very clear at every step what we are proposing so that we can have a 
meaningful discussion.

It's my understanding, for example, that Homebrew allows the user to request to 
install the git master of a particular Homebrew formula, in effect allowing the 
user to (try to) install a version of the software that was never previously 
attempted by the formula's creator. This is what I was afraid was part of what 
was being proposed, and something I would not support adding to MacPorts.


> The examples were meant to illustrate the fact that it's possible to restrict 
> the dependency versions when installing an old package. By specifying maximum 
> versions for all of a package's dependencies (by using something like <=2.4.0 
> or <2.4.0), it would guarantee that the old package would never have to be 
> built against new dependencies, because if a user were to install the old 
> package, it would also install older versions of the dependencies.

I understand that it would be possible to define a syntax for specifying 
dependency versions and that MacPorts base could be enhanced to parse it and 
even to find and download the right version of the portfile and to install 
different versions of ports to different locations.

The problem is that if you are currently compatible with the latest version of 
a dependency, you do not know in advance whether you will be compatible with 
the next version of that dependency.

So you can either be conservative or liberal. MacPorts has tended to be liberal 
in these matters, assuming that a port can build on any architecture, any OS 
version, with any compiler. Once it is discovered that it can't build on some 
subset of those, restrictions can be added. supported_archs can be used to rule 
out incompatible architectures. compiler.blacklist can be used to rule out 
incompatible compilers. Ports can indicate which OS versions they're compatible 
with, though not as elegantly as I would like.

If we continue the liberal approach with dependencies, then we begin from the 
assumption that a port can use any version of a dependency. Let's say libpng 
declares it can use any version of zlib. Then years later zlib 2 is released 
and it is incompatible, breaking all versions of libpng we've published. 
There's now a lot of work to go edit each previous libpng to indicate that it 
requires zlib < 2.

Alternatively, suppose we use the conservative approach and declare that libpng 
requires zlib < 2. Or zlib < 1.3. Or zlib < 1.2.12. We have no way to know in 
what future version of zlib, if any, incompatibility might arise. Suppose we 
declare compatibility with zlib < 2, and years later zlib 2 comes along and 
maintains perfect backward compatibility with zlib 1. Then our libpng would 
continue to use the old zlib 1, even though it would work with zlib 2, thus 
making users use outdated dependencies unnecessarily and possibly exposing them 
to bugs that have already been fixed in newer versions. And the solution then 
is to do a lot of work to go edit each previous libpng to update its zlib 
compatibility list.

Either way, it's signing all port maintainers up for a lot of work. (Many port 
maintainers already don't do the currently expected amount of work to maintain 
their ports.) And all for the benefit of other ports being able to declare that 
they require an old version of libpng. We did have this problem once before, 
when some old ports were not compatible with libpng 1.4 when it came out. 
Rather than introducing a libpng12 compatibility port, we did the right thing 
and fixed other software to be compatible with libpng 1.4. If we had used a 
libpng12 port instead, we likely would never have noticed if and when those 
ports gained libpng 1.4 compatibility and would be forcing libpng12 on users 
unnecessarily for years to come.


> This is one place where MacPort's activate/deactivate would really shine: If 
> someone wanted to install an old package, MacPorts would first deactivate all 
> of the dependencies that were too new, and then install the old versions of 
> the dependencies that are 

Re: Package managers and package versioning

[Jason Liu]

>
> Where by "this" you again mean the ability to specify a
> previously-available version, not an arbitrary version that was never in a
> portfile.
>

Yes, of course. If you look at the folder for the Blender package on the
Debian repo

http://deb.debian.org/debian/pool/main/b/blender/

from my previous example, there are many versions of Blender that were
skipped. For instance, they skipped version 2.82 and went from 2.79b right
to 2.82a. And they skipped versions 2.83.0-2.83.4, and went from 2.82a
right to 2.83.5. Obviously, there would be no way for a user to try to
install Blender version 2.83.2 using apt-get, because that version was
never packaged (i.e. spec files don't exist) for that particular version of
Blender.

Those links again do not explain how to overcome the challenges of building
> an old portfile against new dependencies or on new operating systems or
> compilers.
>

The examples were meant to illustrate the fact that it's possible to
restrict the dependency versions when installing an old package. By
specifying maximum versions for all of a package's dependencies (by using
something like <=2.4.0 or <2.4.0), it would guarantee that the old package
would never have to be built against new dependencies, because if a user
were to install the old package, it would also install older versions of
the dependencies.

This is one place where MacPort's activate/deactivate would really shine:
If someone wanted to install an old package, MacPorts would first
deactivate all of the dependencies that were too new, and then install the
old versions of the dependencies that are compatible with the old package.

Despite the bloat it might work for that use case, but some npm module
> developers unnecessarily restrict the versions of dependencies they're
> compatible with, meaning that you get an unnecessarily outdated version of
> that dependency, even though a newer version of the dependency would work
> and might fix some bug you're experiencing. And this ballooning of
> dependencies can use a tremendously larger amount of disk space than if
> just a single compatible copy of each dep had been installed. We already
> have users complaining about unnecessary dependencies and unnecessary disk
> usage; this would just make it worse.
>

> And do you really want to install a separate copy of gettext for each
> dependency that uses it?
>

Well, let's be frank here. This is the "Apple way". Other than the system
frameworks, Apple has always strongly encouraged that software keep all of
its bits and pieces inside of its application bundle in the /Applications
folder. This has traditionally meant that multiple apps would have their
own copy of the same gettext library in each of their application bundles;
because Apple has never said "hey developers, put all of your shared
libraries in this /usr/local or /opt/local folder". Instead, they've always
gone the route of "each and every app only gets to play in its own sandbox;
you should bring all of your own toys (i.e. libraries) to your own
sandbox... no sharing of toys! (other than the ones Apple provides and
controls, i.e. system frameworks)"

I'm not saying that I agree with Apple's viewpoint, but I do find it a bit
odd that users would use a *nix-like argument of "share as much as possible
to reduce bloat" to complain about something (MacPorts) that lives on an
Apple Corporation-made system, where "share as little as possible" is the
norm. MacPorts is already a shining example of sharing and reusing the same
libraries on a system that highly discourages such behavior.

But you haven't addressed how you could build historical "Portfiles" on a
> newer OS or compiler version when they would not have been written to
> account for the new restrictions imposed by that newer OS or compiler
> version, and we go back to my supposition that a portfile maintainer would
> be asked to keep updating and patching ever old versions of their portfile.
>

How would you address the "implicit declaration of function" problem I
> mentioned? A new version of Xcode, 12, has come along and shown us that a
> zillion of our ports, such as php, do not include the headers that declare
> the functions they are using. We must fix this for compatibility with
> future Macs.
>

>
This is just an example of the type of problem that comes up all the time.
> Yosemite required tons of patches because many build systems misinterpreted
> 10.10 as 10.1. macOS 11 is requiring tons of patches because many build
> systems expected the macOS major version to remain at 10 forever. Many
> build systems need patches now for ARM support. etc. etc. etc.
>

In this particular case, I believe that the Xcode version, and even the
macOS version, should be counted as just another type of dependency... one
that happens to be required in every single portfile. If these (required)
variables also had the ability to specify version ranges, we could
effectively cap each version of a portfile with a maximum compatible

Re: Package managers and package versioning

[Ryan Schmidt]

> On Oct 3, 2020, at 20:23, Jason Liu wrote:
> 
>> On Fri, Oct 2, 2020 at 9:00 PM Ryan Schmidt wrote:
>> 
>> On Oct 2, 2020, at 17:42, Lothar Haeger wrote:
>>> Instead of creating separate copies of perl for each version, it would've 
>>> probably been smarter to fix the limitation in MacPorts that made this 
>>> workaround necessary, i.e. its inability to maintain and install all but 
>>> the latest version of a port. RPM can do it, DEB can do it, MSI can do it, 
>>> nothing unusual in the grand scheme of package managers in general to be 
>>> able to choose a specific version to install. Just MacPorts did not 
>>> implement it yet and when the necessity arose a seemingly simple workaround 
>>> was chosen instead of solving the underlying problem.
>> 
>> I have no familiarity with rpm, deb, msi, or other package managers so I 
>> cannot say whether or how they allow the user to select which version of a 
>> package to install.
> 
> Many modern package management systems give both package makers and users the 
> ability to specify version numbers, both for packages and for dependencies. I 
> call them "modern", because either they have only existed for, say, 10 years 
> or less, and were created from the very beginning with version-choosing 
> capabilities, or if they have been around for around 20 years or more, then 
> they gained the ability one of two ways. Either the ability was added as part 
> of a complete rewrite of the package management system, or it was added as a 
> major new feature.
> 
> One of the most visible consequences of this ability is demonstrated by how 
> dependencies are specified. Giving users the ability to choose which version 
> of a package to install could very easily result in dependency hell. Thus, to 
> prevent this, dependencies also need to be able to have version numbers 
> specified:
>   • RPM Guide: Specifying the Version of the Dependencies
>   • ServerFault: RPM: Set Required: somepackage >= 0.5.0 AND sompackage < 
> 0.6.0
> This results in package management systems with spec files that have 
> dependencies specified like this:
> 
> Requires: somepackage >= 0.5.0, somepackage < 0.6.0

Yup, once it has been shown how it would be possible to install multiple 
versions of a port simultaneously and request specific versions, it is obvious 
that dependencies would need to be able to specify versions. What specific 
syntax would be used to do that isn't important at this point.


>> I'm speaking of the user being able to specify an arbitrary version. If 
>> you're instead thinking that the port maintainer would specify a list of 
>> valid versions or something, that might be more feasible, but still not 
>> without some of the above problems.
> 
> APT has the ability to do this. APT is a package management system used by 
> Debian, Ubuntu, and other flavors of Linux that use .deb-based packages. To 
> repeat what I just linked to, using apt-get, a user can install a specific 
> version of a package:
> 
> apt-get install apache2=2.2.20-1ubuntu1
> 
> YUM can also do this. YUM is a package management system used by Red Hat 
> Enterprise Linux, CentOS, and other flavors of Linux that use RPM-based 
> packages. To repeat what I just linked to, using yum:
> 
> yum install firefox-31.5.3-3.el7_1.x86_64

Where by "this" you again mean the ability to specify a previously-available 
version, not an arbitrary version that was never in a portfile.

Those links again do not explain how to overcome the challenges of building an 
old portfile against new dependencies or on new operating systems or compilers.


> NPM and Yarn, which are PMSes for JavaScript packages, were created with 
> package versioning capabilities as a feature right from their very beginnings:

npm is one I'm familiar with from an attempt to redesign the MacPorts web site 
several years ago. Its intended use is that you install the modules that are 
needed for your web site project. Each of your projects has its own 
node_modules directory and its own copy of every module. Not only that, but 
each module that you install with npm might have its own dependencies which get 
installed in yet another node_modules directory. This can quickly balloon out 
of control as you get 25 different versions of the same dependency. Despite the 
bloat it might work for that use case, but some npm module developers 
unnecessarily restrict the versions of dependencies they're compatible with, 
meaning that you get an unnecessarily outdated version of that dependency, even 
though a newer version of the dependency would work and might fix some bug 
you're experiencing. And this ballooning of dependencies can use a tremendously 
larger amount of disk space than if just a single compatible copy of each dep 
had been installed. We already have users complaining about unnecessary 
dependencies and unnecessary disk usage; this would just make it worse.

Specifying what versions of dependencies are compatible might work in npm 

Package managers and package versioning

[Jason Liu]

On Fri, Oct 2, 2020 at 9:00 PM Ryan Schmidt  wrote:

>
> On Oct 2, 2020, at 17:42, Lothar Haeger wrote:
>
>> Instead of creating separate copies of perl for each version, it would've
>> probably been smarter to fix the limitation in MacPorts that made this
>> workaround necessary, i.e. its inability to maintain and install all but
>> the latest version of a port. RPM can do it, DEB can do it, MSI can do it,
>> nothing unusual in the grand scheme of package managers in general to be
>> able to choose a specific version to install. Just MacPorts did not
>> implement it yet and when the necessity arose a seemingly simple workaround
>> was chosen instead of solving the underlying problem.
>>
>
> I have no familiarity with rpm, deb, msi, or other package managers so I
> cannot say whether or how they allow the user to select which version of a
> package to install.
>

Many modern package management systems give both package makers and users
the ability to specify version numbers, both for packages and for
dependencies. I call them "modern", because either they have only existed
for, say, 10 years or less, and were created from the very beginning with
version-choosing capabilities, or if they have been around for around 20
years or more, then they gained the ability one of two ways. Either the
ability was added as part of a complete rewrite of the package management
system, or it was added as a major new feature.

One of the most visible consequences of this ability is demonstrated by how
dependencies are specified. Giving users the ability to choose which
version of a package to install could very easily result in dependency
hell. Thus, to prevent this, dependencies also need to be able to have
version numbers specified:

   - RPM Guide: Specifying the Version of the Dependencies
   

   - ServerFault: RPM: Set Required: somepackage >= 0.5.0 AND sompackage <
   0.6.0
   


This results in package management systems with spec files that have
dependencies specified like this:

Requires: somepackage >= 0.5.0, somepackage < 0.6.0

I'm speaking of the user being able to specify an arbitrary version. If
> you're instead thinking that the port maintainer would specify a list of
> valid versions or something, that might be more feasible, but still not
> without some of the above problems.
>

APT has the ability to do this.  APT
is a package management system used by Debian, Ubuntu, and other flavors of
Linux that use .deb-based packages. To repeat what I just linked to, using
apt-get, a user can install a specific version of a package:

apt-get install apache2=2.2.20-1ubuntu1

YUM can also do this.

YUM is a package management system used by Red Hat Enterprise Linux,
CentOS, and other flavors of Linux that use RPM-based packages. To repeat
what I just linked to, using yum:

yum install firefox-31.5.3-3.el7_1.x86_64

NPM and Yarn, which are PMSes for JavaScript packages, were created with
package versioning capabilities as a feature right from their very
beginnings:

   - Yarn: Version ranges
   
   - NPM/Greenkeeper: Introduction to Version Ranges
   
   - NPM: Specifying dependencies and devDependencies in a package.json file
   


This results in the possibility of restricting dependencies to very
specific version numbers, like this:

1.4.0-1.5.2 || >=1.5.6 <2.4.0 : Matches version 1.4.0 through 1.5.2, and
1.5.7, but not 1.5.3, 1.5.4, or 2.4.0

I can't currently remember which one at the moment, but I seem to vaguely
recall that the syntax for a certain package manager looked something like:

 install apache2 @2.2.20-1ubuntu1

which, it seems, looks very close to a syntax that would fit MacPorts. For
example, perhaps something like this might be a nice syntax if such a
feature were to be added to MacPorts:

port install vlc @5.0.6 +jack +svg +shout

and in portfiles, something like this for dependency version ranges might
make sense:

depends_lib port:boost >=1.67.0 +cmake_scripts +python37|python38
-mpich_devel \
port:openexr >=1.7.4_3,<2.7.1_2

(this example also includes a capability on my personal wish-list: the
ability to specify a dependency to be installed with/without a certain set
of variants, but without needing to resort to things like using the
active_variants PortGroup or require_active_variants)

As for MacPorts, it's not that we haven't implemented it because we're
> lazy. It's because, besides being