Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Bryan Drewery]

On 12/18/2016 4:31 PM, Baptiste Daroussin wrote:
> Hi all,
> 
> I have been working for a while on 2 long standing feature request for the 
> ports
> tree: flavors and subpackages.
> 
> For flavors I would like to propose a simple approach first which is more 
> like a
> rework of the slave ports for now:
> 
> Examples available here:
> https://reviews.freebsd.org/D8840 (with the implementation)
> and
> https://reviews.freebsd.org/D8843
> 
> Design: introduce a 3rd level in the hierarchy and make it work a bit like 
> slave
> ports
> 
> pros:
> - all slave ports are self hosted under the same directory: easier for
>   maintenance
> - should work with all existing tools
> 
> cons:
> - hackish: it is not really much more than a slave port
> - it adds plenty of new Makefiles :(
> 
> I think anyway this is an improvement
> 
> Next step after that is in would be to extend it to allow some dependency on 
> "I
> depend on whatever flavor if port X"
> 
> Subpackages:
> Design:
> Add a new macro MULTI_PACKAGES
> flag plist with an @pkg{suffixofthesubpackage} file
> the framework will split the plist into small plist and create all the 
> packages
> All variables like COMMENT can be overridden with a 
> COMMENT_${suffixofthesubpackage}
> 
> pros:
> - simple and working almost now
> - allow to simplify lots of ports
> - options friendly (_PACKAGE automatically appends a new entry to
>   MULTI_PACKAGES)
> 
> cons:
> - will break the paradigm that certain tools depend on (portmaster/portupgrade
>   in particular are a huge problem since they are not actively maintained)
> 
> Example of the usage:
> https://people.freebsd.org/~bapt/multipackage.diff
> 
> Note that I took the mpg123 as an example because it was a simple one to test
> not because it may need subpackages
> 
> As a result you build 3 packages:
> mpg123 (the runtime tools)
> mpg123-lib: the runtime libraries
> mpg123-sndio: the sndio plugin
> 
> LIB_DEPENDS on ports depending on libmpg123.so does not have to be changed, 
> the
> framework already automatically register only the mpg123-lib as a dependency 
> and
> not others.
> 
> Not the example is missing one thing: a dependency between mpeg123-lib and
> mpg123
> 
> The second is not ready yet and would take time to land
> 
> Any comment?
> 
> Best regards,
> Bapt
> 

By the way, support for Poudriere on this is progressing.  A Pull
request was submitted with support for another FLAVORS-type
implementation, that pretty much works with your patches as well. I am
working to resolve the DEPENDS_ARGS bug first, which also fixes allowing
the same ORIGIN into the queue multiple times.  Once that is done, then
FLAVORS can be supported in Poudriere.

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Franco Fichtner]

> On 23 Dec 2016, at 10:34 AM, Miroslav Lachman <000.f...@quip.cz> wrote:
> 
> But we don't have that now. For example dns/py-dnspython can create 
> py27-dnspython, py33-dnspython, py34-dnspython, py35-dnspython - four 
> different packages from one origin, one Makefile.

Noticed that too.  This could be easily solved with slave ports that set
this for e.g. py27-dnspython:

USES= python:2.7

and in the master port

USES?= python
USES+= other-uses-stuff

In fact, py-dnspython master port already does it this way.  But most
Python ports do not.


Cheers,
Franco
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Miroslav Lachman]

Baptiste Daroussin wrote on 2016/12/22 21:08:

On Mon, Dec 19, 2016 at 07:12:02PM +0100, Miroslav Lachman wrote:

Matthew Seaman wrote on 2016/12/19 09:45:

On 19/12/2016 07:47, David Demelier wrote:

I have been working for a while on 2 long standing feature request for the ports
tree: flavors and subpackages.

For flavors I would like to propose a simple approach first which is more like a
rework of the slave ports for now:

Examples available here:
https://reviews.freebsd.org/D8840 (with the implementation)
and
https://reviews.freebsd.org/D8843

Design: introduce a 3rd level in the hierarchy and make it work a bit like slave
ports

pros:
- all slave ports are self hosted under the same directory: easier for
maintenance
- should work with all existing tools


This is what I really wanted for years especially for ports like spell
checker. Some are in dedicated categories such as french/aspell while
other are in textproc/-aspell and that's a big mess.

OpenBSD ports has something like textproc/aspell/ and that is
very nice and clean. If the plan is to do the same, that is definitely
a major improvement.



I really like this idea, although it's going to add a lot of extra
directories and very similar small Makefiles to the ports.  Every python
port would grow flavours to support two major versions of python just
for starters, and those additional Makefiles would be almost identical
across the python2 flavour and across the python3 flavour.


Can this be processed by some code in Mk/bsd.*.mk?
I mean if we can add something to the main Makefile then we don't need to
add subdirectories and sub-Makefiles for each Python module port.


If we do that we do break the paradigm: 1 package = 1 origin which will break
portmaster/portupgrade for example


But we don't have that now. For example dns/py-dnspython can create 
py27-dnspython, py33-dnspython, py34-dnspython, py35-dnspython - four 
different packages from one origin, one Makefile.
OK, I noticed now that slave port for py3 was added "recently" but even 
this slave port can create 3 different packages according to what 
default version of python 3 is installed.


Another thing is that portmaster and portupgrade are not well maintained 
and I think ports / ports framework evolution should not be slowed down 
because of "3rd party" utilities. Or we will end up with adding tons of 
directories and small files just to create endless pile of slave ports.


Miroslav Lachman
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Adam Weinberger]

> On 22 Dec, 2016, at 13:04, Baptiste Daroussin  wrote:
> 
> The clean way would be to to just have a new variable in a given port that
> describes the possible variations. But that would break all existing external
> tools that deals with the ports tree. Because they all rely on the fact that
> there is a mapping between a package name and an origin (not that pkg does not
> rely on that.

It's more than just cleaner; it improves the development workflow dramatically. 
Variable-based flavours can be added, modified, and removed easily. c/p/f may 
necessitate recopies and potentially tricky quarterly backports.

Flavours and subpackages are a big deal. I'd prefer that aging, 
non-actively-developed not drive design decisions. I feel like the flavour and 
subpackage omelettes are worth cracking those eggs for.

> So I decided to go another way: add a third level to the ports tree. So far we
> have category/port and I do propose to add a third level: category/port/flavor
> which will keep the paradigm most tools are expected: 1 packagename == 1 
> origin

They're not necessarily redundant: variable-based flavours provide for 
combinations of options, and 3rd-level ports provide a meaningful way to 
categorize nearly-identical ports (like textproc/aspell/{en,fr}). Personally 
I'd love to see both those things happen.

# Adam


-- 
Adam Weinberger
ad...@adamw.org
https://www.adamw.org


___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Baptiste Daroussin]

On Tue, Dec 20, 2016 at 05:41:17PM +0100, Luca Pizzamiglio wrote:
> Hi,
> 
> I think it's a nice to have and an improvement.
> It's quite clean, even if the number of Makefile's can really increase.
> 
> I've some questions:
> 
> Q1) It seems obvious (at least to me), that DOCS and EXAMPLES
> should/could become subpackages.
> How it could be handled by pkg? Are you thinking to add some "magic"
> to enable or disable the automatic installation of specific
> subpackages?

It depends having a subpackage for a bunch of README files is useless while
having a subpackage for ports with loads of html/pdf files would deserve it yes.

Note that there is "magic" planned in pkg at all :)
> 
> Q2) are we opening the door the -devel packages like some Linux distros?

That is an entirely different direction and this would be a policy discussion.

In short:
1/ technically, yes it would allow that
2/ is that what I am aiming at: not at all
> 
> Q3) Do you think there is a general way to decide what should stay an
> OPTION and what should/could become a FLAVOR?

Nope that would be probably mostly driven by common sense
> 
> Q4) Can FLAVORs be in CONFLICT with each others or only conflict-free
> FLAVOR will be accepted?
> If ports can depend to FLAVOR, strange CONFLICTS can arise..

yes flavor can conflict but pkg will detect the conflict and propose to replace
the previous flavor if needed
> 

Best regards,
Bapt


signature.asc
Description: PGP signature

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Baptiste Daroussin]

On Mon, Dec 19, 2016 at 08:25:36PM +0100, Matthieu Volat wrote:
> On Mon, 19 Dec 2016 01:31:43 +0100
> Baptiste Daroussin  wrote:
> 
> > Hi all,
> > 
> > I have been working for a while on 2 long standing feature request for the 
> > ports
> > tree: flavors and subpackages.
> > 
> > For flavors I would like to propose a simple approach first which is more 
> > like a
> > rework of the slave ports for now:
> > 
> > Examples available here:
> > https://reviews.freebsd.org/D8840 (with the implementation)
> > and
> > https://reviews.freebsd.org/D8843
> > 
> > Design: introduce a 3rd level in the hierarchy and make it work a bit like 
> > slave
> > ports
> > 
> > pros:
> > - all slave ports are self hosted under the same directory: easier for
> >   maintenance
> > - should work with all existing tools
> > 
> > cons:
> > - hackish: it is not really much more than a slave port
> > - it adds plenty of new Makefiles :(
> > 
> > I think anyway this is an improvement
> > 
> > Next step after that is in would be to extend it to allow some dependency 
> > on "I
> > depend on whatever flavor if port X"
> > 
> > Subpackages:
> > Design:
> > Add a new macro MULTI_PACKAGES
> > flag plist with an @pkg{suffixofthesubpackage} file
> > the framework will split the plist into small plist and create all the 
> > packages
> > All variables like COMMENT can be overridden with a 
> > COMMENT_${suffixofthesubpackage}
> > 
> > pros:
> > - simple and working almost now
> > - allow to simplify lots of ports
> > - options friendly (_PACKAGE automatically appends a new entry 
> > to
> >   MULTI_PACKAGES)
> > 
> > cons:
> > - will break the paradigm that certain tools depend on 
> > (portmaster/portupgrade
> >   in particular are a huge problem since they are not actively maintained)
> > 
> > Example of the usage:
> > https://people.freebsd.org/~bapt/multipackage.diff
> > 
> > Note that I took the mpg123 as an example because it was a simple one to 
> > test
> > not because it may need subpackages
> > 
> > As a result you build 3 packages:
> > mpg123 (the runtime tools)
> > mpg123-lib: the runtime libraries
> > mpg123-sndio: the sndio plugin
> > 
> > LIB_DEPENDS on ports depending on libmpg123.so does not have to be changed, 
> > the
> > framework already automatically register only the mpg123-lib as a 
> > dependency and
> > not others.
> > 
> > Not the example is missing one thing: a dependency between mpeg123-lib and
> > mpg123
> > 
> > The second is not ready yet and would take time to land
> > 
> > Any comment?
> > 
> > Best regards,
> > Bapt
> 
> Does this approach would manage a file that differ between flavors? Let's say 
> there a libfoo.so file that behave differently wheter an option A is selected 
> or not, but is still present in both cases. 

Yes
> 
> On another note, I kinda liked the macports approach to use the "+" separator 
> regarding naming flavors/options, it allows to better distinguish what in the 
> package name and what are the selected options, and handled itself quite well 
> with multiple instances, like "vim+nls+python+x11"... Did you consider 
> something like that?

No because, actually there were some ports doing that in the past. and we
removed that because it makes it hard to identify programatically packages

Also not that the information about the options used are already stored in the
package and pkg can show them to you

Best regards,
Bapt


signature.asc
Description: PGP signature

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Baptiste Daroussin]

On Mon, Dec 19, 2016 at 01:53:35PM -0500, George Mitchell wrote:
> On 12/18/16 19:31, Baptiste Daroussin wrote:
> > Hi all,
> > 
> > I have been working for a while on 2 long standing feature request for the 
> > ports
> > tree: flavors and subpackages.
> > [...]
> 
> Off topic, I know, but might this eventually lead to FLAVORS for base?
> I would be so grateful to have a SCHED_4BSD flavor of base so I didn't
> have to keep updating my machines from source every time there was a
> security update.  Maybe someone can at least think about it.  -- George
> 

Not at all we are only speaking here on changes in the ports tree that require
no changes at all in pkg

Best regards,
Bapt


signature.asc
Description: PGP signature

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Baptiste Daroussin]

On Mon, Dec 19, 2016 at 07:12:02PM +0100, Miroslav Lachman wrote:
> Matthew Seaman wrote on 2016/12/19 09:45:
> > On 19/12/2016 07:47, David Demelier wrote:
> > > > I have been working for a while on 2 long standing feature request for 
> > > > the ports
> > > > tree: flavors and subpackages.
> > > > 
> > > > For flavors I would like to propose a simple approach first which is 
> > > > more like a
> > > > rework of the slave ports for now:
> > > > 
> > > > Examples available here:
> > > > https://reviews.freebsd.org/D8840 (with the implementation)
> > > > and
> > > > https://reviews.freebsd.org/D8843
> > > > 
> > > > Design: introduce a 3rd level in the hierarchy and make it work a bit 
> > > > like slave
> > > > ports
> > > > 
> > > > pros:
> > > > - all slave ports are self hosted under the same directory: easier for
> > > >maintenance
> > > > - should work with all existing tools
> > > > 
> > > This is what I really wanted for years especially for ports like spell
> > > checker. Some are in dedicated categories such as french/aspell while
> > > other are in textproc/-aspell and that's a big mess.
> > > 
> > > OpenBSD ports has something like textproc/aspell/ and that is
> > > very nice and clean. If the plan is to do the same, that is definitely
> > > a major improvement.
> > > 
> > 
> > I really like this idea, although it's going to add a lot of extra
> > directories and very similar small Makefiles to the ports.  Every python
> > port would grow flavours to support two major versions of python just
> > for starters, and those additional Makefiles would be almost identical
> > across the python2 flavour and across the python3 flavour.
> 
> Can this be processed by some code in Mk/bsd.*.mk?
> I mean if we can add something to the main Makefile then we don't need to
> add subdirectories and sub-Makefiles for each Python module port.

If we do that we do break the paradigm: 1 package = 1 origin which will break
portmaster/portupgrade for example

Bapt


signature.asc
Description: PGP signature

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Baptiste Daroussin]

On Mon, Dec 19, 2016 at 08:45:36AM +, Matthew Seaman wrote:
> On 19/12/2016 07:47, David Demelier wrote:
> >> I have been working for a while on 2 long standing feature request for the 
> >> ports
> >> tree: flavors and subpackages.
> >>
> >> For flavors I would like to propose a simple approach first which is more 
> >> like a
> >> rework of the slave ports for now:
> >>
> >> Examples available here:
> >> https://reviews.freebsd.org/D8840 (with the implementation)
> >> and
> >> https://reviews.freebsd.org/D8843
> >>
> >> Design: introduce a 3rd level in the hierarchy and make it work a bit like 
> >> slave
> >> ports
> >>
> >> pros:
> >> - all slave ports are self hosted under the same directory: easier for
> >>   maintenance
> >> - should work with all existing tools
> >>
> > This is what I really wanted for years especially for ports like spell
> > checker. Some are in dedicated categories such as french/aspell while
> > other are in textproc/-aspell and that's a big mess.
> > 
> > OpenBSD ports has something like textproc/aspell/ and that is
> > very nice and clean. If the plan is to do the same, that is definitely
> > a major improvement.
> > 
> 
> I really like this idea, although it's going to add a lot of extra
> directories and very similar small Makefiles to the ports.  Every python
> port would grow flavours to support two major versions of python just
> for starters, and those additional Makefiles would be almost identical
> across the python2 flavour and across the python3 flavour.
> 
> Is it the intention that -devel or versioned ports are treated as
> flavours as well? So for example we could end up with lang/python/27
> lang/python/33 lang/python/34 and lang/python/35 ?

I think that would be a very very bad idea to have that. There should be only
one version for python2 and one version for python3 no more

> Why can't you have both flavoured and unflavoured variants of the same
> port -- eg. devel/example as well as devel/example/foo and
> devel/example/bar ?

In theory we could I'm working on that direction now.
> 
> How well do flavours and sub-packages combine?  www/nginx would make a
> good example there, given it now has both loadable and compiled-in
> modules plus a couple of slave ports that just build different module
> load-outs.
> 
The should fit well.

I have an example locally with php

I have lang/php56 which is now a single port that create zillions of packages
(the same granularity as we have now in ports) and a flavour lang/php56/zts
which does the same.

Best regards,
Bapt


signature.asc
Description: PGP signature

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Baptiste Daroussin]

On Tue, Dec 20, 2016 at 07:11:11PM +1100, Dewayne Geraghty wrote:
> Thanks Bapt et al,
> 
> I use FreeBSD and the ports system extensively, we build everything from
> source and largely customise approx 25% of the 900 packages we rely upon.
> I'm more than a little concerned to have changes performed against the
> ports infrastructure.  As our primary sources of (whats coming) "Change"
> information are the: Quarterly reports and the OS Release Notes;
> after-the-fact sources are a daily review of
> https://lists.freebsd.org/pipermail/svn-src-stable-11/2016-December/thread.html
> for OS impact; and the excellent Freshports.
> 
> So a few questions:
> 
> Could you be able to enlighten us (the readers) so we can better understand
> what will be changed; or share your vision of the benefits and operational
> impact for operational people that build: from source; and those that only
> use binary?

Sure so there are 2 different things that are requested for a long time by lots
of users:
1/ flavors

This is the avility to say this port should be built with a list of
variations by default. There are 2 kind of flavors:
a/ the one that are not conflicting: for example being able to have all python
   ports available for both python 2 and python 3 by default and at the same
   time. Right now it is not directly possible without a hack
b/ the one that makes variations of a given package right now done a hackish way
   via the (for example) *-nox11, *-static or *-lite.

The goal of the change I do propose aims at making both kind of flavors easily
maintainable.

The difficulty is bringing in that feature without breaking anything for end
users.

The clean way would be to to just have a new variable in a given port that
describes the possible variations. But that would break all existing external
tools that deals with the ports tree. Because they all rely on the fact that
there is a mapping between a package name and an origin (not that pkg does not
rely on that.

It means that for poudriere and synth we can easily adapt them, both are very
actively maintained and their design should allow "easily" to integrate that
But portmaster and portupgrade would be deeply broken as not actively
maintained

So I decided to go another way: add a third level to the ports tree. So far we
have category/port and I do propose to add a third level: category/port/flavor
which will keep the paradigm most tools are expected: 1 packagename == 1 origin

Maybe some tools would have to be updated a bit, but that would be minor patches

With my current patch the only problem I see that the category/port level is
unused if there are flavors while I could certainly make it "the default flavor"

the drawback of this approach is it will add a lots of new small files and
directory.

The most important part of the flavors is probably the ability to provide
natively support for python2 and python3 at the same time

A good side effect of adding a third level is we could now imagine regrouping
some ports (the openbsd ports tree does that already) like aspell where we could
have:
textproc/aspell and textproc/aspell/fr textproc/aspell/en etc which would make
things a bit cleaner

2/ sub packages

sub packages is the ability from one build to create multiple packages.
The goal is to avoid the giant gcc package as a runtime dependency for example
where we could provide a gcc-libs package for the runtime libary

It would also allow to save a lot of building time for things like php or qt We
could end up with on single port (so one single extraction and one single run of
the configure script) while keeping the flexbility of the current split which is
existing right now in the ports tree with zillion of complex slave ports.

The big issue while designing that solution is it cannot be made transparent for
the users as it will for sure break the paradigm: 1 pkgname == 1 origin

> 
> Is there a transition plan or schedule for the bulk of these changes to
> occur?

For flavours it should be transparent if not that would be a bug except if
everyone argue I should break the paradigm 1 pkgname == 1 origin and go for the
clean implementation
> 
> Will the flavors/subpackages be developed separately from the existing
> ports suite?  (I'm hoping that the parent ports will be unaffected, and so
> our existing build procedures continue to build correctly)

I don't see how it can be developped separately, can you elaborate more?
> 
> How will we (the users/admins) track or be informed of changes or better,
> planned/soon changes?  (will changes to ports, particularly parent ports,
> be co-ordinated through UPDATING or perhaps a new FLAVOURS file if the
> parent is say a stub and the real decisions are relocated to slaves?)

Yes of course UPDATING and MOVED when needed
for example shells/bash-static|shells/bash/static|

> 
> Will there be any guidance regarding how flavours/packages should be
> created or the criteria for creating sub-packages (secure/insecure; all
> 

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Christian Schwarz]

On Tue, Dec 20, 2016 at 10:12:04AM +0100, Franco Fichtner wrote:
> And lastly... if we have the automatic "default" flavour that is
> defined by the OPTIONS_DEFAULT knobs, we could finally avoid pkg
> upgrading custom builds by knowing that somebody built a "custom"
> version of their port and that there is no equivalent to upgrade
> to.
> 
> This is exciting!

While it is exciting, I would be sad to see flavours be the solution to
pkg not recognizing build OPTIONS_ as part of a package's identity
right now[1].

What is not entirely clear to me:

  Are flavours always a tuple of values for OPTIONS_ defined by their
  master port?

The reason I bring this up: a binary package is identified by the
following information:

  - pkg name (the master's name, unique over ports tree)
  - version & revision
  - the artifacts used to build the binary
(tarballs, but also build dependencies, ...)
  - a vector of available options
  - a vector of values for the available options
  - (other stuff you could probably find in a talk on reproducible
 builds)

It is obvious that a master port will have *many* binary incarnations.
To my understanding, flavours are a comfortable way to write down some
commonly used incarnations.

Reducing the package manager's job to checking that some incarnation of
the package is present is surely better than no support for this.

However, I think the logical next step is to have ports declare that
they depend on a subset of specific configuration values being used in
their dependencies.

In this scenario, flavours are no different to pkg than self-built
ports with custom-picked non-(flavour|standard)ized options.
This, I would very much prefer.

Either way: big thanks to bapt and those who contributed so far!

-- Christian

[1] Please continue reading for what I understand as 'package identity'.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Luca Pizzamiglio]

Hi,

I think it's a nice to have and an improvement.
It's quite clean, even if the number of Makefile's can really increase.

I've some questions:

Q1) It seems obvious (at least to me), that DOCS and EXAMPLES
should/could become subpackages.
How it could be handled by pkg? Are you thinking to add some "magic"
to enable or disable the automatic installation of specific
subpackages?

Q2) are we opening the door the -devel packages like some Linux distros?

Q3) Do you think there is a general way to decide what should stay an
OPTION and what should/could become a FLAVOR?

Q4) Can FLAVORs be in CONFLICT with each others or only conflict-free
FLAVOR will be accepted?
If ports can depend to FLAVOR, strange CONFLICTS can arise..

Thanks for the great job! I'll keep contributing as much as I can.

Best regards,
Luca

On Tue, Dec 20, 2016 at 10:12 AM, Franco Fichtner  wrote:
>
>> On 20 Dec 2016, at 9:42 AM, Franco Fichtner  wrote:
>>
>> To emphasise on this:
>
> And lastly... if we have the automatic "default" flavour that is
> defined by the OPTIONS_DEFAULT knobs, we could finally avoid pkg
> upgrading custom builds by knowing that somebody built a "custom"
> version of their port and that there is no equivalent to upgrade
> to.
>
> This is exciting!
>
>
> Cheers,
> Franco
> ___
> freebsd-ports@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Franco Fichtner]

> On 20 Dec 2016, at 9:42 AM, Franco Fichtner  wrote:
> 
> To emphasise on this:

And lastly... if we have the automatic "default" flavour that is
defined by the OPTIONS_DEFAULT knobs, we could finally avoid pkg
upgrading custom builds by knowing that somebody built a "custom"
version of their port and that there is no equivalent to upgrade
to.

This is exciting!


Cheers,
Franco
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Franco Fichtner]

> On 20 Dec 2016, at 9:27 AM, Franco Fichtner  wrote:
> 
> We shouldn't use "-" or "/" anyway, should we?  Please no fancy things
> like "~" or so.  No arbitrary package names...

To emphasise on this:

A flavour should act as a full replacement of its unflavoured package, that
means the package name must be kept.  Only one flavour (or unflavoured)
package can be installed at all times.  As an example:

A weird package "foo" requires "vim", but the user doesn't want to deal with
X11, the user should be able to:

# pkg install vim:lite foo

This should not try to change "vim:lite" to "vim".

# pkg install vim

This should be perfectly fine afterwards, too.

Every "vim" should act as "vim", not revoking the integrity of the package
dependency on vim during e.g. pkg upgrade.  No forced install should be
needed to do this as long as the shared libraries and dependencies are still
satisfied.  And maybe the moral of the story is that flavours should not
be depended on by default, although it could be a possibility for special
cases.

This is something that is really really needed.  An very good example would
be Suricata package with Hyperscan right now, where Hyperscan does not work
on all amd64 architectures, so we need to have a replacement package.  But
if that replacement package without Hyperscan needs to be a separate port,
any package depending on Suricata (e.g. a distribution or GUI package) will
complain about the missing dependency and try to undo a Suricata-No-Hyperscan
package[1] as it conflicts and changes back to the defunct package on upgrade.


Cheers,
Franco

[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210490
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Franco Fichtner]

Hi,

> On 19 Dec 2016, at 1:31 AM, Baptiste Daroussin  wrote:
> 
> For flavors I would like to propose a simple approach first which is more 
> like a
> rework of the slave ports for now:

This progression sure is nice to see!  I like "category/portname/flavour"
origin a lot, but how is it handled in terms of packages names?

Are we going to see something like:

# pkg install myport:flavour

We shouldn't use "-" or "/" anyway, should we?  Please no fancy things
like "~" or so.  No arbitrary package names...

In OpenBSD, installing flavoured packages has been hard to script in the
past, offering a prompt whenever the main package is going to be installed.
The thing to think about here is that

# pkg install myport

Should *only* install the default port, especially with -y option.

# pkg install myport:

This *could* prompt for flavours, then.  The nice thing should be the
user doesn't have to care about flavours if that is so.

Flavours as you showed can be very simple.  Why not go the extra mile
here:

FLAVOURS=   sub1 sub2

OPTIONS_sub1=   EXPLICIT LIST OF OPTIONS
OPTIONS_sub2=   ANOTHER LIST OF OPTIONS

And keep everything as is.  No need for sub-packages?  No implied
OPTIONS_DEFAULT, no nothing.  A single line to grep and change.  :)

>From this perspective, nothing changes for users of the ports tree, options
are defined by the main port and all of its flavours are neatly stored in
the Makefile.  People can still use all options during rebuild, even the
ones only used in flavours.


Cheers,
Franco
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Dewayne Geraghty]

Thanks Bapt et al,

I use FreeBSD and the ports system extensively, we build everything from
source and largely customise approx 25% of the 900 packages we rely upon.
I'm more than a little concerned to have changes performed against the
ports infrastructure.  As our primary sources of (whats coming) "Change"
information are the: Quarterly reports and the OS Release Notes;
after-the-fact sources are a daily review of
https://lists.freebsd.org/pipermail/svn-src-stable-11/2016-December/thread.html
for OS impact; and the excellent Freshports.

So a few questions:

Could you be able to enlighten us (the readers) so we can better understand
what will be changed; or share your vision of the benefits and operational
impact for operational people that build: from source; and those that only
use binary?

Is there a transition plan or schedule for the bulk of these changes to
occur?

Will the flavors/subpackages be developed separately from the existing
ports suite?  (I'm hoping that the parent ports will be unaffected, and so
our existing build procedures continue to build correctly)

How will we (the users/admins) track or be informed of changes or better,
planned/soon changes?  (will changes to ports, particularly parent ports,
be co-ordinated through UPDATING or perhaps a new FLAVOURS file if the
parent is say a stub and the real decisions are relocated to slaves?)

Will there be any guidance regarding how flavours/packages should be
created or the criteria for creating sub-packages (secure/insecure; all
options on/off; most useable options on; most liked by the maintainer; most
likely to be used for a datacentre; most likely to be used for desktops;
...)?   Will "The Porter's Handbook" be updated for things like criteria;
naming conventions etc?

For folks (like me) that build entirely from source and customise options
to build the applications, how will flavours/subpackages be of benefit?
Will the ability to customise ports, as they exist today, remain?  Will I
even notice a change?

I'd like to plan ahead to make this transition seemless and continue to use
FreeBSD and the excellent ports system as we do now.

I started with FreeBSD 2.2.8.  There were packages available from the
FreeBSD website.  It was a terrific aid.  We also enjoyed the different
flavours of jail that were provided by ezjail.  However over time, both
evolved as did our expertise to customise our ports (~200 custom ports) and
Jamie Gratton evolved the jail system to eliminate our need of the
excellent ezjail tool.  So I can see merit in, what very little I'm
guessing of, the next evolution of ports.

Aside: we already build different package configurations from existing
ports' source.  (eg different bind910 with/without kerberos; different
samba44's; simultaineous building of dhcp-[server|client|relay] etc)

I look forward to being on the same page and to understand where this is
going, the likely/potential impact; the naming conventions; etc.

Kind regards, Dewayne.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Matthew Seaman]

On 19/12/2016 18:53, George Mitchell wrote:
> On 12/18/16 19:31, Baptiste Daroussin wrote:
>> Hi all,
>>
>> I have been working for a while on 2 long standing feature request for the 
>> ports
>> tree: flavors and subpackages.
>> [...]
> 
> Off topic, I know, but might this eventually lead to FLAVORS for base?
> I would be so grateful to have a SCHED_4BSD flavor of base so I didn't
> have to keep updating my machines from source every time there was a
> security update.  Maybe someone can at least think about it.  -- George

I believe you can already build packages for several different kernel
configurations.

Cheers,

Matthew




signature.asc
Description: OpenPGP digital signature

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Matthieu Volat]

On Mon, 19 Dec 2016 01:31:43 +0100
Baptiste Daroussin  wrote:

> Hi all,
> 
> I have been working for a while on 2 long standing feature request for the 
> ports
> tree: flavors and subpackages.
> 
> For flavors I would like to propose a simple approach first which is more 
> like a
> rework of the slave ports for now:
> 
> Examples available here:
> https://reviews.freebsd.org/D8840 (with the implementation)
> and
> https://reviews.freebsd.org/D8843
> 
> Design: introduce a 3rd level in the hierarchy and make it work a bit like 
> slave
> ports
> 
> pros:
> - all slave ports are self hosted under the same directory: easier for
>   maintenance
> - should work with all existing tools
> 
> cons:
> - hackish: it is not really much more than a slave port
> - it adds plenty of new Makefiles :(
> 
> I think anyway this is an improvement
> 
> Next step after that is in would be to extend it to allow some dependency on 
> "I
> depend on whatever flavor if port X"
> 
> Subpackages:
> Design:
> Add a new macro MULTI_PACKAGES
> flag plist with an @pkg{suffixofthesubpackage} file
> the framework will split the plist into small plist and create all the 
> packages
> All variables like COMMENT can be overridden with a 
> COMMENT_${suffixofthesubpackage}
> 
> pros:
> - simple and working almost now
> - allow to simplify lots of ports
> - options friendly (_PACKAGE automatically appends a new entry to
>   MULTI_PACKAGES)
> 
> cons:
> - will break the paradigm that certain tools depend on (portmaster/portupgrade
>   in particular are a huge problem since they are not actively maintained)
> 
> Example of the usage:
> https://people.freebsd.org/~bapt/multipackage.diff
> 
> Note that I took the mpg123 as an example because it was a simple one to test
> not because it may need subpackages
> 
> As a result you build 3 packages:
> mpg123 (the runtime tools)
> mpg123-lib: the runtime libraries
> mpg123-sndio: the sndio plugin
> 
> LIB_DEPENDS on ports depending on libmpg123.so does not have to be changed, 
> the
> framework already automatically register only the mpg123-lib as a dependency 
> and
> not others.
> 
> Not the example is missing one thing: a dependency between mpeg123-lib and
> mpg123
> 
> The second is not ready yet and would take time to land
> 
> Any comment?
> 
> Best regards,
> Bapt

Does this approach would manage a file that differ between flavors? Let's say 
there a libfoo.so file that behave differently wheter an option A is selected 
or not, but is still present in both cases. 

On another note, I kinda liked the macports approach to use the "+" separator 
regarding naming flavors/options, it allows to better distinguish what in the 
package name and what are the selected options, and handled itself quite well 
with multiple instances, like "vim+nls+python+x11"... Did you consider 
something like that?


-- 
Matthieu Volat 



pgpKyNhOOomNS.pgp
Description: OpenPGP digital signature

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Guido Falsi]

On 12/19/16 19:53, George Mitchell wrote:

On 12/18/16 19:31, Baptiste Daroussin wrote:

Hi all,

I have been working for a while on 2 long standing feature request for the ports
tree: flavors and subpackages.
[...]


Off topic, I know, but might this eventually lead to FLAVORS for base?
I would be so grateful to have a SCHED_4BSD flavor of base so I didn't
have to keep updating my machines from source every time there was a
security update.  Maybe someone can at least think about it.  -- George


While you wait for the improvement you ask, you can use binary updates 
and just recompile the kernel by hand.


Should spare you some time and effort.

--
Guido Falsi 
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[George Mitchell]

On 12/18/16 19:31, Baptiste Daroussin wrote:
> Hi all,
> 
> I have been working for a while on 2 long standing feature request for the 
> ports
> tree: flavors and subpackages.
> [...]

Off topic, I know, but might this eventually lead to FLAVORS for base?
I would be so grateful to have a SCHED_4BSD flavor of base so I didn't
have to keep updating my machines from source every time there was a
security update.  Maybe someone can at least think about it.  -- George

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Miroslav Lachman]

Matthew Seaman wrote on 2016/12/19 09:45:

On 19/12/2016 07:47, David Demelier wrote:

I have been working for a while on 2 long standing feature request for the ports
tree: flavors and subpackages.

For flavors I would like to propose a simple approach first which is more like a
rework of the slave ports for now:

Examples available here:
https://reviews.freebsd.org/D8840 (with the implementation)
and
https://reviews.freebsd.org/D8843

Design: introduce a 3rd level in the hierarchy and make it work a bit like slave
ports

pros:
- all slave ports are self hosted under the same directory: easier for
   maintenance
- should work with all existing tools


This is what I really wanted for years especially for ports like spell
checker. Some are in dedicated categories such as french/aspell while
other are in textproc/-aspell and that's a big mess.

OpenBSD ports has something like textproc/aspell/ and that is
very nice and clean. If the plan is to do the same, that is definitely
a major improvement.



I really like this idea, although it's going to add a lot of extra
directories and very similar small Makefiles to the ports.  Every python
port would grow flavours to support two major versions of python just
for starters, and those additional Makefiles would be almost identical
across the python2 flavour and across the python3 flavour.


Can this be processed by some code in Mk/bsd.*.mk?
I mean if we can add something to the main Makefile then we don't need 
to add subdirectories and sub-Makefiles for each Python module port.


Miroslav Lachman
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Kyle Evans]

On Mon, Dec 19, 2016 at 2:45 AM, Matthew Seaman  wrote:
> Why can't you have both flavoured and unflavoured variants of the same
> port -- eg. devel/example as well as devel/example/foo and
> devel/example/bar ?

It seems like it would make sense to allow devel/example to be a
default flavor so that, for instantiated example, editors/vim-lite =>
editors/vim/lite and editors/vim could potentially be a 'full' flavor
or unflavored, if that's your flavor. I personally learn towards
default flavor, though, because that gives you a chance to be slightly
more descriptive.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[Matthew Seaman]

On 19/12/2016 07:47, David Demelier wrote:
>> I have been working for a while on 2 long standing feature request for the 
>> ports
>> tree: flavors and subpackages.
>>
>> For flavors I would like to propose a simple approach first which is more 
>> like a
>> rework of the slave ports for now:
>>
>> Examples available here:
>> https://reviews.freebsd.org/D8840 (with the implementation)
>> and
>> https://reviews.freebsd.org/D8843
>>
>> Design: introduce a 3rd level in the hierarchy and make it work a bit like 
>> slave
>> ports
>>
>> pros:
>> - all slave ports are self hosted under the same directory: easier for
>>   maintenance
>> - should work with all existing tools
>>
> This is what I really wanted for years especially for ports like spell
> checker. Some are in dedicated categories such as french/aspell while
> other are in textproc/-aspell and that's a big mess.
> 
> OpenBSD ports has something like textproc/aspell/ and that is
> very nice and clean. If the plan is to do the same, that is definitely
> a major improvement.
> 

I really like this idea, although it's going to add a lot of extra
directories and very similar small Makefiles to the ports.  Every python
port would grow flavours to support two major versions of python just
for starters, and those additional Makefiles would be almost identical
across the python2 flavour and across the python3 flavour.

Is it the intention that -devel or versioned ports are treated as
flavours as well? So for example we could end up with lang/python/27
lang/python/33 lang/python/34 and lang/python/35 ?

Why can't you have both flavoured and unflavoured variants of the same
port -- eg. devel/example as well as devel/example/foo and
devel/example/bar ?

How well do flavours and sub-packages combine?  www/nginx would make a
good example there, given it now has both loadable and compiled-in
modules plus a couple of slave ports that just build different module
load-outs.

Matthew




signature.asc
Description: OpenPGP digital signature

Re: HEADSUP: FLAVORS (initial version) and subpackages proposals

[David Demelier]

2016-12-19 1:31 GMT+01:00 Baptiste Daroussin :
> Hi all,
>
> I have been working for a while on 2 long standing feature request for the 
> ports
> tree: flavors and subpackages.
>
> For flavors I would like to propose a simple approach first which is more 
> like a
> rework of the slave ports for now:
>
> Examples available here:
> https://reviews.freebsd.org/D8840 (with the implementation)
> and
> https://reviews.freebsd.org/D8843
>
> Design: introduce a 3rd level in the hierarchy and make it work a bit like 
> slave
> ports
>
> pros:
> - all slave ports are self hosted under the same directory: easier for
>   maintenance
> - should work with all existing tools
>

This is what I really wanted for years especially for ports like spell
checker. Some are in dedicated categories such as french/aspell while
other are in textproc/-aspell and that's a big mess.

OpenBSD ports has something like textproc/aspell/ and that is
very nice and clean. If the plan is to do the same, that is definitely
a major improvement.

> cons:
> - hackish: it is not really much more than a slave port
> - it adds plenty of new Makefiles :(
>
> I think anyway this is an improvement
>
> Next step after that is in would be to extend it to allow some dependency on 
> "I
> depend on whatever flavor if port X"
>
> Subpackages:
> Design:
> Add a new macro MULTI_PACKAGES
> flag plist with an @pkg{suffixofthesubpackage} file
> the framework will split the plist into small plist and create all the 
> packages
> All variables like COMMENT can be overridden with a 
> COMMENT_${suffixofthesubpackage}
>
> pros:
> - simple and working almost now
> - allow to simplify lots of ports
> - options friendly (_PACKAGE automatically appends a new entry to
>   MULTI_PACKAGES)
>

Also lovely as it will probably remove some packages that duplicate a
lot the ports tree (-nox11 for example). If I understood correctly
that one.

> Any comment?

Yes, make it real :)

And again, thanks for your amazing and regular good work bapt!

-- 
Demelier David
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

HEADSUP: FLAVORS (initial version) and subpackages proposals

[Baptiste Daroussin]

Hi all,

I have been working for a while on 2 long standing feature request for the ports
tree: flavors and subpackages.

For flavors I would like to propose a simple approach first which is more like a
rework of the slave ports for now:

Examples available here:
https://reviews.freebsd.org/D8840 (with the implementation)
and
https://reviews.freebsd.org/D8843

Design: introduce a 3rd level in the hierarchy and make it work a bit like slave
ports

pros:
- all slave ports are self hosted under the same directory: easier for
  maintenance
- should work with all existing tools

cons:
- hackish: it is not really much more than a slave port
- it adds plenty of new Makefiles :(

I think anyway this is an improvement

Next step after that is in would be to extend it to allow some dependency on "I
depend on whatever flavor if port X"

Subpackages:
Design:
Add a new macro MULTI_PACKAGES
flag plist with an @pkg{suffixofthesubpackage} file
the framework will split the plist into small plist and create all the packages
All variables like COMMENT can be overridden with a 
COMMENT_${suffixofthesubpackage}

pros:
- simple and working almost now
- allow to simplify lots of ports
- options friendly (_PACKAGE automatically appends a new entry to
  MULTI_PACKAGES)

cons:
- will break the paradigm that certain tools depend on (portmaster/portupgrade
  in particular are a huge problem since they are not actively maintained)

Example of the usage:
https://people.freebsd.org/~bapt/multipackage.diff

Note that I took the mpg123 as an example because it was a simple one to test
not because it may need subpackages

As a result you build 3 packages:
mpg123 (the runtime tools)
mpg123-lib: the runtime libraries
mpg123-sndio: the sndio plugin

LIB_DEPENDS on ports depending on libmpg123.so does not have to be changed, the
framework already automatically register only the mpg123-lib as a dependency and
not others.

Not the example is missing one thing: a dependency between mpeg123-lib and
mpg123

The second is not ready yet and would take time to land

Any comment?

Best regards,
Bapt


signature.asc
Description: PGP signature