Re: This is going to break port building without poudriere!

[Alexander Leidinger]

Am 2024-01-30 13:24, schrieb Luca Pizzamiglio:

Hi Alexander,

On Tue, Jan 30, 2024 at 11:31 AM Alexander Leidinger <alexan...@leidinger.net> wrote:

Am 2024-01-30 10:42, schrieb Luca Pizzamiglio:

Hi Alexander.

Subpackage modularization of existing ports (i.e. qt6-tools) provides
benefits to package users/builders: smaller dependency footprint,
smaller usage on disk (useful for embedded systems), smaller jails.
There are no benefits nor regressions for port builders for this
specific use case.

only for leaf ports. Ports which are a dependency can not converted to
subpackages (in the sense of "no slave port available for the
subpackages").

Nicely worded. I agree to that. At the same time it is a regression for port builders. Installing from ports is not on par with installing from pkg anymore in this case. What you describe means we can use subpackages

I disagree that this is regression, but I agree that it's an unexpected change.

And at the same time you talk about a breaking change in the next part.

Installing from ports can install more than installing from pkg.

However, it introduces a divergence that requires more attention.

I don't call it a regression, as it doesn't introduce bugs, ports can be successfully built as before.

A divergence which requires more attention is a nice way of telling it is a regression.

On the other hand, moving master/slave ports to subpackages comes with the cost of losing the ability to build the slave ports independently.

For package users there is no change and some benefit for package
builders.
The issue can be mitigated by introducing options to select which

single subpackage cannot be built independently.

subpackage to build (available for make install as well), but, still, a

specific example for this particular case where it is not a proper
mitigation.

This is not a proper mitigation. I used mail/roundcube and lang/php as a

It's a mitigation, as you can reduce what you need to build, but it's not a solution and it doesn't fix the breaking change. It only offers a way to install less, but no way to install a subpackage only.

This limitation _can_ be unacceptable in some cases. For example,

the master/slave approach will stay.

bofh@, the php maintainer, considers subpackages not the way to go, so

Do you see that you just told that one of the best role models for
subpackages according to you previous mail will not use subpackages
because of the limitations I highlighted?

Yes, your arguments contributed to change my opinion on this point. Is this a problem?

No.

We introduced subpackages in the framework to explore all use cases, by

trying and testing its adoption.
By doing so we have already identified some issues (i.e. USES is not
subpackage-aware yet) and interesting new use cases (subpackage with
debug symbols).

You are surely aware that you haven't mentioned one of the points I
talked about as an issue, don't you? I have not seen any technical
answer which shows that my technical description of issues is wrong. I

not want to fix them" (I understand it like that because I see no
acknowledgement of those issues, just answers which look like a
smokescreen and pointing into directions of good faith). If you would
acknowledge the technical issues highlighted in this thread (or show
evidence that those regressions can not happen) and tell that the

I talk about, my understanding of the situation would change.

want to make you aware that I understand the responses from you and the others in this thread as: "we do not care about those regressions and do adoption of subpackages is monitored to not introduce those regressions

Those seem serious accusations and they trouble me.

So you are saying that my answers are "smokescreen and pointing into directions of good faith". Am I trying to deceive you? Are you assuming that I have an ill intent that needs to be masked?

I do not think you try to deceive me or anyone else. I do not think you have ill intents. When we met some years ago in Essen I got a favorable impression about you (I remember you as smart and nice) and this has not changed (and you can be assured that I'm not angry or similar, and that you can imagine my "voice" in this thread as calm as in my interaction with everyone back then in Essen). What I "accuse" (to pick up one of your words, I rather call it "giving the impression in your answers") you is that you postulate that subpackages as implemented does not create issues or play down the issues when converting master/slave ports to subpackages, or implicitly tell that "make install" is not worth it / something you support / ... and people shall use packages instead (I use packages myself in most cases, but sometimes "make install" is the most pragmatic solution, and then users shall not fall into a trap... "make install" is an asset, not a burden).

As per: "we don't care about those regressions and do not want to fix them". I've already acknowledged that subpackages are going to introduce a breaking change.

It's possible to have different outcomes if you use packages and ports.

We always had different outcomes regarding build dependencies. Having those differences is not the point. The point is that this change will cripple the ports collection. Converting a port to make use of subpackages removes the possibility to have fine grained control over what is installed which is exposed to a sensible security context (aka the internet). Yes, sorry, I'm riding on the lang/php example to death... it's a good example.

We are going to have a portmgr meeting soon and we will see how to proceed.

I'm looking forward to the outcome.

The solutions you propose (subpackge+slave port) don't make any sense to me, I would prefer to focus on a long term solution.

It doesn't make sense to me either, but is the only way the current implementation of subpackages will not cripple the functionality of "make install" in the mid-term. I also would prefer a long term solution. Namely making subpackages first class items for pkg (= no different naming -> s/~/-/, and having the origin recorded inside the metadata), and having the possibility to target subpackages for a make install (as a quick idea: introduce "make install TARGET_SUBPACKAGES=a,b,c", have the port build as needed (full or only the subpackage), build the subpackage out of the plist, and only install the subpackage with pkg).

Technically, it should be possible to install a portion of a port.

It's more difficult to build only a part of a port, but this aspect is less relevant.

Only building the subpackage is an optimization problem (for lang/php this is possible, for other ports maybe not), not a problem of a crippled feature. Yes, it would be nice if it is possible (if upstream allows it) to build subpackages, but this is not something I would ask portmgr to policy. But I ask portmgr to policy the security relevant features, as an example "make install" of mail/roundcube (or take any other port which doesn't require lang/php but falls into the same category of security implications).

As per master/slave merge use case, we will let the maintainers decide

if they want to move forward with subpackage adoptions, knowing the

regression for port builders, but we won't push them in this direction.

I consider it unfortunate that you describe it like that. I was hoping
you would tell that portmgr will prevent the removal of slave packages
and enforce the rule of having slave ports for subpackages aware ports
to prevent regressions for users which install from ports (until the
technical valid issues I have pointed out are resolved -- and they can
be fixed, a first step would be to make the names of subpackages match

prevent churn later).


lock/policy by portmgr on the subpackages feature like I already
described in my previous mails (until the regressions the use of
subpackages would create are fixed). This would allow for more
experimentation by a lot of people without the need to patch the ports

ports with a lot of dependencies.

normal packages names = replacing the '~' in the name with a '-' ASAP to Note, in src some big discussion like this of having several committers identify regressions and no immediate fix would lead to a backout until it is fixed. I do not ask for a backout of this, but I ask for a strong tree and without introducing regressions for a simple "make install" of

Are you asking to completely block the introduction of subpackages? A complete block seems excessive to me, but some additional restriction can be considered.

I ask portmgr to

 - not accept the removal of slave ports for existing subpackages.

- not accept conversions of existing ports to subpackages which remove slave ports (and as such touches the dependencies of other ports). - not accept new ports with subpackages without slave ports to be able to target the subpackages for "make install". All this until the implementation of subpackages is fixed in a way that we can make lang/php a subpackages aware port and do not install more dependencies in "make install" in mail/roundcube. Again, lang/php serves here as an example of a perfect-fit port for subpackages, and at the same time a perfect example of the issues we get on "make install". So no, I do not ask to block it completely. I simply ask to not remove features. I'm aware that this introduces a burden on package building. If those which operate a package building infrastructure want to have some more restrictions because of this, then this is off course at the discretion of them to speak up. Personally I'm willing to take the hit for my poudriere runs, if this allows to improve the subpackages feature in the tree. I additionally suggest to rethink the naming scheme of subpackages (replacing the '~' with a '-' to have the same filename on disk for the slave ports and the corresponding subpackage) before new subpackages aware ports are introduced. In my opinion this facilitates future updates / improvements to the current implementation of subpackages. I'm open to discuss specifics, but I suggest to open a new thread for this... in case there is interest to discuss it.

In any case, an update on the subpackage topic needs to be shared in the next few days.

Bye,
Alexander.

I'm looking forward to it. I consider subpackages a very nice feature in the general case. I have simply seen some bugs/issues/regressions in the current implementation which make me believe it is not yet ready to let us use it without the above mentioned governance until those issues are fixed.

Best regards,
Luca

Bye,
Alexander.

Best regards,
Luca

On Mon, Jan 29, 2024 at 10:04 AM Alexander Leidinger
<alexan...@leidinger.net> wrote:

Am 2024-01-27 16:59, schrieb Luca Pizzamiglio:

Hi Stefan.

Let's start from the beginning, as it seems that things are not
clear.

Subpackages is a feature to create multiple packages from one build
Those subpackages can depend on the main package.
The main package cannot depend on any subpackages.
This limits the cases where subpackages can be applied. The main
package MUST be independent from its subpackages. Subpackages can
only
add features (like a plugin).
To recall your NLS example (ref
https://reviews.freebsd.org/D16457#715443): this is not a use-case
for
subpackages. If the main program/library needs to be compiled

If a port needs to be built multiple times to create different
subpackages, this is not a viable case for subpackages.
A good candidate is qt6-tools: this port contains multiple tools
(designer, linguist, help, and so on). Those tools could be put in
different subpackages.

I hope this explanation helps to clarify and address some of your
concerns.

differently with or without NLS, this is not viable for subpackages.

As I read this, lang/php is the best example of a port where
subpackages
has a benefit (in the sense it matches your description above to the
letter, the main port independent from the subpacakges, and what can
be
build as subpackages is a plugin/extension).

OPTIONS and SUBPACKAGES
Do we have to convert OPTIONS to SUBPACKAGES? No.
Can a SUBPACKAGE be built only if an OPTION is enabled? Yes.
The only viable use cases for SUBPACKAGES being enabled/disabled by
OPTIONS is limited to those portions of the port that do not affect
the
main package.


Consolidate master/slave ports in one bigger ports
About this topic, I guess your concerns are mainly related to
potential
explosion of build time of the consolidated ports.
This is a justified concern.
In those cases, we are suggesting to convert slave ports in
SUBPACKAGES
enabled via OPTIONS.
This will allow port builders to configure the bigger ports to not

the previous build time.


the consolidation makes sense or not.
If a consolidation is going to result more problematic than
beneficial,
it can be reverted and subpackages not adopted for the use case.

Examples are: additional libraries, additional data files, and so on. build all SUBPACKAGES, but only the needed ones. This should restore However, as for the php case, the maintainer is going to evaluate if

If I understand the sum of all the above correct, you suggest to
remove

terms of the current implementation of subpackages). Or worded
differently to the same effect (as I only care about the effect and
not
the intention), when someone converts a port to subpackages, the

ports shall not be introduced in this case).

Removing slave ports means we can not depend upon specific parts
anymore

security implications of to much being installed and active if
installed
via make install in /usr/ports/something/with_webinterface. -> best

to
use subpackages / shows what is a regression in the features we
provide
in our ports collection.

While qt6-tools may be a good example where subpackages makes sense,
we
can not depend on subpacakges for "make install", and as if the port
would be converted to have subpackages but no slave ports are

installed files.

slave ports and to use subpackages instead (where this makes sense in corresponding slave packages shall be removed (or for new ports: slave when installing from ports, as the subpackages can not be targeted for install directly and my example of a subpackages aware php results in example of lang/php to use subpacakges is the best example of why not introduced, pkg install and make install would differ in the amount of

For port builders

Port builders can experience longer build times in the future, as

If this is the case, the larger ports should provide OPTIONS to not
build unneeded subpackages.
If no OPTION is available, please work with the maintainer to
introduce
one.

master/slave ports could be consolidated in one single larger ports.

I fail to see the benefit:
We either lose the possibility to target parts of a package (when
slave
ports are removed / not introduced) on make install (with a similar
amount of build time for the ports tree as it is right now), or get
higher build times for the package builders. In both cases we do not
gain something significant.


ports (there is the case of py39-rpds-py failing to build in my

a
runtime issue in maturin, which shows up in my the cross build on
amd64-intel for amd64-athlon64... which in the end leads me to build
py-rpds-py on the target machine from ports), the current
implementation

for each subpackage.


which don't come with a slave port.

I would be OK to lift this restriction, when the current
implementation
is changed to be able to only install the files of a subpacakge on
make
install (an implementation could be to require "make install
TARGET_SUBPACAKGES=sub1,sub2,sub3" or "make install-subpackage1

according to this requirement, those people designing, implementing
and
reviewing this can argue about such details).

Keeping the current implementation (with the restriction of always

that a slave port covers a specific subpackage which would allow
package

have the same package name as the slave port, no idea if this has a

same package name, but it surely would be confusing for people at
first
look).


benefit, only drawbacks. Can you please provide an example of a
benefit
I fail to see (yes, more modularity for qt6-tools may be beneficial
for
some people, but the cost/benefit between qt6-tools (something which
we
don't provide right now) and "make install" (what we provide right
now)
or "build time reduction for package builders" (which would have a
benefit for a lot of use cases) is in my books much more in the
direction of "make install" and "build time reduction" than towards
the
modularization of qt6-tools)?

Bye,
Alexander.

--
http://www.Leidinger.net alexan...@leidinger.net: PGP
0x8F31830F9F2772BF
http://www.FreeBSD.org    netch...@freebsd.org  : PGP
0x8F31830F9F2772BF

If we want to keep the (useful in some cases) feature to install from poudriere which I tried to debug with the author of py-maturin due to of subpackages has to come with the requirement to create slave ports That's my concern, and that's the reason why I have the opinion, that portmgr has to keep the lock on subpackages and reject any subpackage install-subpacakge2"... as long as recursive dependencies are handled introducing slave packages for subpackages) would need a way to denote builders to skip those slave ports (and the subpacakges would need to technical disadvantage in terms of having 2 different origins for the TLDR: for the use cases you specified in the beginning, I do not see a

--

http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netch...@freebsd.org : PGP 0x8F31830F9F2772BF

--
http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.org    netch...@freebsd.org  : PGP 0x8F31830F9F2772BF

signature.asc
Description: OpenPGP digital signature