Re: harder and harder to avoid pkg

[Torsten Zuehlsdorff]

On 16.10.2016 05:16, Alfred Perlstein wrote:

Has anyone actually looked/asked how other OS's solve this problem?


Yes, for various linux distributions. This provided me with so many 
reasons to stay and work with the ports-tree.



I too found "xxx-dev" vs "xxx-lib" annoying until I realized how clean
it actually is.


You need to be more clear with that. Most distributions already fail 
with the "normal" names of modules. You need to install php-xml to get 
UTF8 and also you directly get DOM, which has nothing to do with it. 
Also simpleXML is installed with it, which is a lightweight replacement 
for DOM and the other XML functions so you do not use more than one of 
it - but of course everything is installed.
php-dev? Yes, it is definitely needed to get the PostgreSQL module for 
PHP. But for MySQL there is php-mysql.


In FreeBSD? php-xml, php-simplexml, php-dom, php-pg, php-mysql.

I'm working as a developer and as an administrator. As administrator it 
is fairly easy to get a list of modules and software to install and have 
packages named like the software.


I've created a sheet of translations for our linux-admins. "If developer 
wants this $software you need to install $something-different".



We should definitely be surveying the landscape before rolling our own
NIH solution.


We do. And the ports-tree offers features missing in most others 
distributions. pkg audit for example. Or a consistent placement of the 
files and configurations.


Greetings,
Torsten
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Alfred Perlstein]

Has anyone actually looked/asked how other OS's solve this problem?

I too found "xxx-dev" vs "xxx-lib" annoying until I realized how clean 
it actually is.


We should definitely be surveying the landscape before rolling our own 
NIH solution.


-Alfred

On 10/14/16 8:30 AM, Julian Elischer wrote:

On 14/10/2016 4:27 AM, Matthieu Volat wrote:

On Fri, 14 Oct 2016 13:05:35 +0200
David Demelier  wrote:


2016-10-14 11:22 GMT+02:00 Baptiste Daroussin :

It is imho doable in both sides.

We could imagine tagging the plist/manifest so pkg can allow a user 
to install
only the things tagged as runtime for exemple which would do the 
job. for what
Julian is asking for beside adding lots of complexity pkg(8) and 
adding a

nightmare in the solver.

That would "please" the people that want "hey keep the giant flat 
package as it
is better for dev given I don't have to install the -devel version 
something"

and the people wanting fine grain selection if they need to.

But on the ports side that would be a nightmare having to tag all 
the plist (and

this cannot be automated because there are to many corner cases.

IIRC, rpm builders have script that automate this by finding files in
standard directories. Probably by checking in the stage a include/
directory and "tag" it as the development part.
Unless things changed very recently, not quite : you have to pile 
subpackage declaration and files sections according to the 
subpackages you create. The only things it has to ease the burden is 
you can use wildcard patterns to select files.



It will be the most smart way of doing this but still require some
addition to pkg. Probably like:

- pkg install mylib
- pkg install -t dev mylib
- pkg install -t runtime mylib
- pkg install -t dev,runtime,doc mylib

Just thinking ;)
More options, then more options to `pkg info` to get what was 
installed when something cannot build, then more pkg search options 
and manpage because more "-t" flags will be added and we don't know 
what's needed?



I'm glad people are at least thinking about it...

I don't think there are so many categories.  Are we installing onto a 
development machine, user machine, or an appliance? appliances don't 
need man pages. User machines need man pages for programs but not for 
libraries and developer machines.. everything..



___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"



___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Dave Horsfall]

On Fri, 14 Oct 2016, David Demelier wrote:

> It's not writing a port that is complicated, it's the whole 
> infrastructure.

You should see the Macports infrastructure...  Fairly easy for the end 
user, but those developers sweat blood to make it so.

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Julian Elischer]

On 14/10/2016 4:27 AM, Matthieu Volat wrote:

On Fri, 14 Oct 2016 13:05:35 +0200
David Demelier  wrote:


2016-10-14 11:22 GMT+02:00 Baptiste Daroussin :

It is imho doable in both sides.

We could imagine tagging the plist/manifest so pkg can allow a user to install
only the things tagged as runtime for exemple which would do the job. for what
Julian is asking for beside adding lots of complexity pkg(8) and adding a
nightmare in the solver.

That would "please" the people that want "hey keep the giant flat package as it
is better for dev given I don't have to install the -devel version something"
and the people wanting fine grain selection if they need to.

But on the ports side that would be a nightmare having to tag all the plist (and
this cannot be automated because there are to many corner cases.

IIRC, rpm builders have script that automate this by finding files in
standard directories. Probably by checking in the stage a include/
directory and "tag" it as the development part.

Unless things changed very recently, not quite : you have to pile subpackage 
declaration and files sections according to the subpackages you create. The 
only things it has to ease the burden is you can use wildcard patterns to 
select files.


It will be the most smart way of doing this but still require some
addition to pkg. Probably like:

- pkg install mylib
- pkg install -t dev mylib
- pkg install -t runtime mylib
- pkg install -t dev,runtime,doc mylib

Just thinking ;)

More options, then more options to `pkg info` to get what was installed when something 
cannot build, then more pkg search options and manpage because more "-t" flags 
will be added and we don't know what's needed?


I'm glad people are at least thinking about it...

I don't think there are so many categories.  Are we installing onto a 
development machine, user machine, or an appliance? appliances don't 
need man pages. User machines need man pages for programs but not for 
libraries and developer machines.. everything..



___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Matthieu Volat]

On Fri, 14 Oct 2016 13:05:35 +0200
David Demelier  wrote:

> 2016-10-14 11:22 GMT+02:00 Baptiste Daroussin :
> > It is imho doable in both sides.
> >
> > We could imagine tagging the plist/manifest so pkg can allow a user to 
> > install
> > only the things tagged as runtime for exemple which would do the job. for 
> > what
> > Julian is asking for beside adding lots of complexity pkg(8) and adding a
> > nightmare in the solver.
> >
> > That would "please" the people that want "hey keep the giant flat package 
> > as it
> > is better for dev given I don't have to install the -devel version 
> > something"
> > and the people wanting fine grain selection if they need to.
> >
> > But on the ports side that would be a nightmare having to tag all the plist 
> > (and
> > this cannot be automated because there are to many corner cases.
> 
> IIRC, rpm builders have script that automate this by finding files in
> standard directories. Probably by checking in the stage a include/
> directory and "tag" it as the development part.

Unless things changed very recently, not quite : you have to pile subpackage 
declaration and files sections according to the subpackages you create. The 
only things it has to ease the burden is you can use wildcard patterns to 
select files.

> It will be the most smart way of doing this but still require some
> addition to pkg. Probably like:
> 
> - pkg install mylib
> - pkg install -t dev mylib
> - pkg install -t runtime mylib
> - pkg install -t dev,runtime,doc mylib
> 
> Just thinking ;)

More options, then more options to `pkg info` to get what was installed when 
something cannot build, then more pkg search options and manpage because more 
"-t" flags will be added and we don't know what's needed?

-- 
Matthieu Volat 



pgpZY_aqAJ_bU.pgp
Description: OpenPGP digital signature

Re: harder and harder to avoid pkg

[Kurt Jaeger]

Hi!

> > This is an appliance class machine. It has 2G of storage and that has
> > to  include 2 copies for the OS so we can ping-pong for upgrades.
> 
> I can get > 2GB CPU cache per system (spread over 8+ sockets) these 
> days. Is it really reasonable to expect port maintainers to take up the 
> work and classify their maintained ports for you to save you an 
> additional 2GB of cheap flash storage?

Letting the appliance-market slip away to other platforms should be
avoided.

> At a certain scale those 
> trade-offs might make sense for you, but I suspect most FreeBSD port 
> maintainers and FreeBSD users don't mind a few 100 kB of documentation 
> and headers on their systems. Aren't there easier solutions which don't 
> require a lot of manual work?

Using the pkg-plist of packages by removing those files not in
bin/ or lib/ might solve approx. 80% of the problem. Someone's
willing to test this 8-} ?

-- 
p...@opsec.eu+49 171 3101372 4 years to go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Jan Bramkamp]

On 14/10/2016 09:39, Julian Elischer wrote:

On 13/10/2016 10:33 AM, RW via freebsd-ports wrote:

On Tue, 11 Oct 2016 11:59:47 -0700
Julian Elischer wrote:


As the number of dependencies between packages get ever higher, it
becomes more and more difficult to compile packages and the
dependence on binary precompiled packages is increased. However
binary packages are unsuitable for some situations.  We really need
to follow the lead of some of the Linux groups and have -runtime and
-devel versions of packages,  OR  we what woudlbe smarter, woudl be
to have several "sub manifests" to allow unpacking in different
environments.


A simple example:   libxml2

This package installs include files and libraries and dicumentation
etc.

yet if I build an appliance , I want it to only install a singe file.

/usr/local/lib/libxml2.so.2

What practical problem does installing the include files and man pages
cause you?

I have to delete them from the appliance I'm building up.
So I need to get the manifest, remove the files I want from it, and
delete every other file mentioned.

This is an appliance class machine. It has 2G of storage and that has
to  include 2 copies for the OS so we can ping-pong for upgrades.


I can get > 2GB CPU cache per system (spread over 8+ sockets) these 
days. Is it really reasonable to expect port maintainers to take up the 
work and classify their maintained ports for you to save you an 
additional 2GB of cheap flash storage? At a certain scale those 
trade-offs might make sense for you, but I suspect most FreeBSD port 
maintainers and FreeBSD users don't mind a few 100 kB of documentation 
and headers on their systems. Aren't there easier solutions which don't 
require a lot of manual work?


 * Documentation and source code compresses well. Can you use a 
read-only lzma or gzip compressed filesystem with GEOM uncompress?


 * Can you use snapshots (and rerooting) to rollback failed updates 
instead of keeping two full copies around?


-- Jan Bramkamp
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[David Demelier]

2016-10-14 11:22 GMT+02:00 Baptiste Daroussin :
> It is imho doable in both sides.
>
> We could imagine tagging the plist/manifest so pkg can allow a user to install
> only the things tagged as runtime for exemple which would do the job. for what
> Julian is asking for beside adding lots of complexity pkg(8) and adding a
> nightmare in the solver.
>
> That would "please" the people that want "hey keep the giant flat package as 
> it
> is better for dev given I don't have to install the -devel version something"
> and the people wanting fine grain selection if they need to.
>
> But on the ports side that would be a nightmare having to tag all the plist 
> (and
> this cannot be automated because there are to many corner cases.

IIRC, rpm builders have script that automate this by finding files in
standard directories. Probably by checking in the stage a include/
directory and "tag" it as the development part.

It will be the most smart way of doing this but still require some
addition to pkg. Probably like:

- pkg install mylib
- pkg install -t dev mylib
- pkg install -t runtime mylib
- pkg install -t dev,runtime,doc mylib

Just thinking ;)

-- 
Demelier David
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Matthew Seaman]

On 10/14/16 10:22, Baptiste Daroussin wrote:
> We could imagine tagging the plist/manifest so pkg can allow a user to install
> only the things tagged as runtime for exemple which would do the job. for what
> Julian is asking for beside adding lots of complexity pkg(8) and adding a
> nightmare in the solver.
> 
> That would "please" the people that want "hey keep the giant flat package as 
> it
> is better for dev given I don't have to install the -devel version something"
> and the people wanting fine grain selection if they need to.
> 
> But on the ports side that would be a nightmare having to tag all the plist 
> (and
> this cannot be automated because there are to many corner cases.

You still need something like this whichever way sub-packages are
implemented -- compiling and staging the port generates a whole load of
files and you somehow have to identify each of them as docs, examples,
whatever either for tagging in the plist or for turning into sub-packages.

Some of that you can do heuristically, but yeah -- this classification
job would be a thing that port maintainers get to enjoy.

It should be possible to create meta-packages that do nothing except
depend on commonly used combinations of sub-packages as a convenience
for people installing software at the command line.  For example one
that could have the same overall result as installing an all-in-one
package at the moment.  I believe something like this is planned for the
base system packages.

> Having the port that grows the feature would be really nice because no work
> would be needed on pkg :) and that would reduce cluster package building as we
> could merge qt, php etc into one port that builds multiple sub packages.

True, that would save a number of repetitive compilations.  Of course,
what you save by implementing sub-packages you'ld immediately lose (and
more) by implementing package flavours.

Cheers,

Matthew



signature.asc
Description: OpenPGP digital signature

Re: harder and harder to avoid pkg

[Baptiste Daroussin]

On Fri, Oct 14, 2016 at 09:54:07AM +0200, Mathieu Arnold wrote:
> Le 14/10/2016 à 09:34, Julian Elischer a écrit :
> > On 13/10/2016 5:42 AM, David Demelier wrote:
> >> 2016-10-12 10:04 GMT+02:00 Andrea Venturoli :
> >>> On 10/12/16 09:24, Matthieu Volat wrote:
> >>>
>  And GNU/Linuxes can be a PITA when you have to track -dev(el) packages
>  (which sometimes really requires -bin, -app or whatever), or worst,
>  describe
>  to people how they are supposed to build your software with weird
>  subpackage
>  names.
> 
>  I really like that ports provides the software project as intended by
>  upstream (modulo options).
> >>>
> >>> Just a "me too" here!
> >> Could not agree more.
> >>
> >> Please forget that idea.
> >>
> >> I just hate having to install libfoo, libfoo-dev, libfoo-dbg,
> >> libfoo-doc, libfoo-whatever each time I need to develop on Linux.
> >> Please do not transform FreeBSD as a Linux distribution :)
> >>
> >> I love the way FreeBSD and some very sparse Linux distributions
> >> provide the packages exactly how it would be installed by hand (=
> >> vanilla).
> >>
> >> FreeBSD offers some options and very few changes for better
> >> integration but packages are provided vanilla. You want a package? You
> >> install /packagename/ nothing more, nothing less. I really would like
> >> to see simple vanilla packages for the next 10 years.
> >>
> >> The FreeBSD ports is already extremely complicated, do not make it
> >> even harder :(
> > The suggestion is not for ports, but for packages..
> > a single package could be unpacked in 'runtime only' or 'everything'
> > mode.
> > basically one package, two manifests.  So no "foo-devel" or "foo-runtime"
> > just 'foo'
> 
> It is for ports, because packages are built using ports, and ports would
> need to grow the feature.
> 

It is imho doable in both sides.

We could imagine tagging the plist/manifest so pkg can allow a user to install
only the things tagged as runtime for exemple which would do the job. for what
Julian is asking for beside adding lots of complexity pkg(8) and adding a
nightmare in the solver.

That would "please" the people that want "hey keep the giant flat package as it
is better for dev given I don't have to install the -devel version something"
and the people wanting fine grain selection if they need to.

But on the ports side that would be a nightmare having to tag all the plist (and
this cannot be automated because there are to many corner cases.

Having the port that grows the feature would be really nice because no work
would be needed on pkg :) and that would reduce cluster package building as we
could merge qt, php etc into one port that builds multiple sub packages.

Best regards,
Bapt


signature.asc
Description: PGP signature

Re: harder and harder to avoid pkg

[Mathieu Arnold]

Le 14/10/2016 à 09:34, Julian Elischer a écrit :
> On 13/10/2016 5:42 AM, David Demelier wrote:
>> 2016-10-12 10:04 GMT+02:00 Andrea Venturoli :
>>> On 10/12/16 09:24, Matthieu Volat wrote:
>>>
 And GNU/Linuxes can be a PITA when you have to track -dev(el) packages
 (which sometimes really requires -bin, -app or whatever), or worst,
 describe
 to people how they are supposed to build your software with weird
 subpackage
 names.

 I really like that ports provides the software project as intended by
 upstream (modulo options).
>>>
>>> Just a "me too" here!
>> Could not agree more.
>>
>> Please forget that idea.
>>
>> I just hate having to install libfoo, libfoo-dev, libfoo-dbg,
>> libfoo-doc, libfoo-whatever each time I need to develop on Linux.
>> Please do not transform FreeBSD as a Linux distribution :)
>>
>> I love the way FreeBSD and some very sparse Linux distributions
>> provide the packages exactly how it would be installed by hand (=
>> vanilla).
>>
>> FreeBSD offers some options and very few changes for better
>> integration but packages are provided vanilla. You want a package? You
>> install /packagename/ nothing more, nothing less. I really would like
>> to see simple vanilla packages for the next 10 years.
>>
>> The FreeBSD ports is already extremely complicated, do not make it
>> even harder :(
> The suggestion is not for ports, but for packages..
> a single package could be unpacked in 'runtime only' or 'everything'
> mode.
> basically one package, two manifests.  So no "foo-devel" or "foo-runtime"
> just 'foo'

It is for ports, because packages are built using ports, and ports would
need to grow the feature.

-- 
Mathieu Arnold




signature.asc
Description: OpenPGP digital signature

Re: harder and harder to avoid pkg

[Julian Elischer]

On 13/10/2016 10:33 AM, RW via freebsd-ports wrote:

On Tue, 11 Oct 2016 11:59:47 -0700
Julian Elischer wrote:


As the number of dependencies between packages get ever higher, it
becomes more and more difficult to compile packages and the
dependence on binary precompiled packages is increased. However
binary packages are unsuitable for some situations.  We really need
to follow the lead of some of the Linux groups and have -runtime and
-devel versions of packages,  OR  we what woudlbe smarter, woudl be
to have several "sub manifests" to allow unpacking in different
environments.


A simple example:   libxml2

This package installs include files and libraries and dicumentation
etc.

yet if I build an appliance , I want it to only install a singe file.

/usr/local/lib/libxml2.so.2

What practical problem does installing the include files and man pages
cause you?

I have to delete them from the appliance I'm building up.
So I need to get the manifest, remove the files I want from it, and 
delete every other file mentioned.


This is an appliance class machine. It has 2G of storage and that has 
to  include 2 copies for the OS so we can ping-pong for upgrades.



___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"



___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Julian Elischer]

On 13/10/2016 5:42 AM, David Demelier wrote:

2016-10-12 10:04 GMT+02:00 Andrea Venturoli :

On 10/12/16 09:24, Matthieu Volat wrote:


And GNU/Linuxes can be a PITA when you have to track -dev(el) packages
(which sometimes really requires -bin, -app or whatever), or worst, describe
to people how they are supposed to build your software with weird subpackage
names.

I really like that ports provides the software project as intended by
upstream (modulo options).


Just a "me too" here!

Could not agree more.

Please forget that idea.

I just hate having to install libfoo, libfoo-dev, libfoo-dbg,
libfoo-doc, libfoo-whatever each time I need to develop on Linux.
Please do not transform FreeBSD as a Linux distribution :)

I love the way FreeBSD and some very sparse Linux distributions
provide the packages exactly how it would be installed by hand (=
vanilla).

FreeBSD offers some options and very few changes for better
integration but packages are provided vanilla. You want a package? You
install /packagename/ nothing more, nothing less. I really would like
to see simple vanilla packages for the next 10 years.

The FreeBSD ports is already extremely complicated, do not make it
even harder :(

The suggestion is not for ports, but for packages..
a single package could be unpacked in 'runtime only' or 'everything' mode.
basically one package, two manifests.  So no "foo-devel" or "foo-runtime"
just 'foo'



Regards,



___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[David Demelier]

2016-10-14 8:14 GMT+02:00 Loïc BLOT :
> FreeBSD ports are complicated ?
> Does someone of you tryed to do a Debian package, it's even more
> complicated as you should modify many path, split package in multiple
> packages, do the service engineering with systemV or systemD, etc ?

It's not writing a port that is complicated, it's the whole infrastructure.

-- 
Demelier David
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Loïc BLOT]

FreeBSD ports are complicated ?
Does someone of you tryed to do a Debian package, it's even more
complicated as you should modify many path, split package in multiple
packages, do the service engineering with systemV or systemD, etc ?
-- 
Best regards,
Loïc BLOT, 
UNIX systems, security and network engineer
http://www.unix-experience.fr


Le jeudi 13 octobre 2016 à 18:08 +0200, Miroslav Lachman a écrit :
> David Demelier wrote on 2016/10/13 14:42:
> > 2016-10-12 10:04 GMT+02:00 Andrea Venturoli :
> > > On 10/12/16 09:24, Matthieu Volat wrote:
> > > 
> > > > And GNU/Linuxes can be a PITA when you have to track -dev(el)
> > > > packages
> > > > (which sometimes really requires -bin, -app or whatever), or
> > > > worst, describe
> > > > to people how they are supposed to build your software with
> > > > weird subpackage
> > > > names.
> > > > 
> > > > I really like that ports provides the software project as
> > > > intended by
> > > > upstream (modulo options).
> > > 
> > > 
> > > Just a "me too" here!
> > 
> > Could not agree more.
> > 
> > Please forget that idea.
> > 
> > I just hate having to install libfoo, libfoo-dev, libfoo-dbg,
> > libfoo-doc, libfoo-whatever each time I need to develop on Linux.
> > Please do not transform FreeBSD as a Linux distribution :)
> > 
> > I love the way FreeBSD and some very sparse Linux distributions
> > provide the packages exactly how it would be installed by hand (=
> > vanilla).
> > 
> > FreeBSD offers some options and very few changes for better
> > integration but packages are provided vanilla. You want a package?
> > You
> > install /packagename/ nothing more, nothing less. I really would
> > like
> > to see simple vanilla packages for the next 10 years.
> > 
> > The FreeBSD ports is already extremely complicated, do not make it
> > even harder :(
> 
> +1 for this!
> 
> Miroslav Lachman
> 
> ___
> freebsd-ports@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.o
> rg"
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[RW via freebsd-ports]

On Tue, 11 Oct 2016 11:59:47 -0700
Julian Elischer wrote:

> As the number of dependencies between packages get ever higher, it 
> becomes more and more difficult to compile packages and the
> dependence on binary precompiled packages is increased. However
> binary packages are unsuitable for some situations.  We really need
> to follow the lead of some of the Linux groups and have -runtime and
> -devel versions of packages,  OR  we what woudlbe smarter, woudl be
> to have several "sub manifests" to allow unpacking in different
> environments.
> 
> 
> A simple example:   libxml2
> 
> This package installs include files and libraries and dicumentation
> etc.
> 
> yet if I build an appliance , I want it to only install a singe file.
> 
> /usr/local/lib/libxml2.so.2

What practical problem does installing the include files and man pages
cause you?
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Miroslav Lachman]

David Demelier wrote on 2016/10/13 14:42:

2016-10-12 10:04 GMT+02:00 Andrea Venturoli :

On 10/12/16 09:24, Matthieu Volat wrote:


And GNU/Linuxes can be a PITA when you have to track -dev(el) packages
(which sometimes really requires -bin, -app or whatever), or worst, describe
to people how they are supposed to build your software with weird subpackage
names.

I really like that ports provides the software project as intended by
upstream (modulo options).



Just a "me too" here!


Could not agree more.

Please forget that idea.

I just hate having to install libfoo, libfoo-dev, libfoo-dbg,
libfoo-doc, libfoo-whatever each time I need to develop on Linux.
Please do not transform FreeBSD as a Linux distribution :)

I love the way FreeBSD and some very sparse Linux distributions
provide the packages exactly how it would be installed by hand (=
vanilla).

FreeBSD offers some options and very few changes for better
integration but packages are provided vanilla. You want a package? You
install /packagename/ nothing more, nothing less. I really would like
to see simple vanilla packages for the next 10 years.

The FreeBSD ports is already extremely complicated, do not make it
even harder :(


+1 for this!

Miroslav Lachman

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[David Demelier]

2016-10-12 10:04 GMT+02:00 Andrea Venturoli :
> On 10/12/16 09:24, Matthieu Volat wrote:
>
>> And GNU/Linuxes can be a PITA when you have to track -dev(el) packages
>> (which sometimes really requires -bin, -app or whatever), or worst, describe
>> to people how they are supposed to build your software with weird subpackage
>> names.
>>
>> I really like that ports provides the software project as intended by
>> upstream (modulo options).
>
>
> Just a "me too" here!

Could not agree more.

Please forget that idea.

I just hate having to install libfoo, libfoo-dev, libfoo-dbg,
libfoo-doc, libfoo-whatever each time I need to develop on Linux.
Please do not transform FreeBSD as a Linux distribution :)

I love the way FreeBSD and some very sparse Linux distributions
provide the packages exactly how it would be installed by hand (=
vanilla).

FreeBSD offers some options and very few changes for better
integration but packages are provided vanilla. You want a package? You
install /packagename/ nothing more, nothing less. I really would like
to see simple vanilla packages for the next 10 years.

The FreeBSD ports is already extremely complicated, do not make it
even harder :(

Regards,

-- 
Demelier David
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Kubilay Kocak]

On 12/10/2016 8:12 PM, Matthew Seaman wrote:
> On 2016/10/12 09:43, Kubilay Kocak wrote:
>>> You are describing the 'sub-packages' concept that has been
>>> knocking
 around for some time.  With sub-packages you'ld divide up the
 result of staging each port into various chunks:
>> Yep, like this:
>> 
>> Mar 6 2016 - https://reviews.freebsd.org/D5563 Ports framework
>> "variants" proof-of-concept (with poudriere support)
>> 
>> Status Report Dec 2015 - Supporting Variants in the Ports
>> Framework
>> 
>> https://www.freebsd.org/news/status/report-2015-10-2015-12.html#Supporting-Variants-in-the-Ports-Framework
>
>> 
> Variants is a related but different concept -- known as 'flavours'
> (or 'flavors') in some parts.  The difference is that 'sub packages'
> divide up the output from one compilation of the sources, whereas
> 'variants' or 'flavours' require the same source code to be
> recompiled with different options.  (As you'ld need to do to create
> eg. py27- and py34- versions of python modules.) Both are things we'd
> like to have in ports, but they can be implemented pretty much
> separately.

They could be, but they don't need to be. From one perspective, division
of a port (or package) is exactly a 'variant'.

Yes a 'part' (-debug package) of a whole (-full package) is a "sub
package", but the 'debug' variant of the foo port only includes debug
files, and has a -debug suffix.

Variants (in its current PoC form) is just a generic implementation for
enabling one-to-many-packages, and is not prescriptive. The 'what to
Vary: on' (like the HTTP headers), including perhaps what to include or
exclude from the package, is left as a exercise for the configuration,
or portmgr or a later stage discussion.

There is nothing explicitly prescribing that specified 'variants' must
compile source each time (or do anything else specific for that matter),
or that said variants cannot merely execute the "dividing up" (on some
basis) logic on the resulting artifacts that were created in the
common/base 'variant'.

> Cheers,
> 
> Matthew
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Vlad K.]

On 2016-10-12 10:27, Julian Elischer wrote:

what I really need is a RUNTIME option that produces a package with
only those files needed to satisfy external run-time depdencies, or
the actual demands of the user itself. However since those files are


Right. But then the question is how do you define what is minimum 
required code to satisfy external run time deps? Can the framework 
assume it by path, or should the maintainers define it through options?


IMHO if the latter, then my suggestion is not to define any roles 
("runtime") but define groups of files that can then be included in 
whatever role. Eg, all include/... files be switched with HEADERS 
option. Many ports already do DOCS, MANPAGES, EXAMPLES. So that's a set 
criteria one can base -runtime variant upon:


OPTIONS_UNSET= HEADERS DOCS MANPAGES EXAMPLES


--

Vlad K.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Matthew Seaman]

On 2016/10/12 09:43, Kubilay Kocak wrote:
>> You are describing the 'sub-packages' concept that has been knocking 
>> > around for some time.  With sub-packages you'ld divide up the result
>> > of staging each port into various chunks:
> Yep, like this:
> 
> Mar 6 2016 - https://reviews.freebsd.org/D5563
> Ports framework "variants" proof-of-concept (with poudriere support)
> 
> Status Report Dec 2015 - Supporting Variants in the Ports Framework
> 
> https://www.freebsd.org/news/status/report-2015-10-2015-12.html#Supporting-Variants-in-the-Ports-Framework

Variants is a related but different concept -- known as 'flavours' (or
'flavors') in some parts.  The difference is that 'sub packages' divide
up the output from one compilation of the sources, whereas 'variants' or
'flavours' require the same source code to be recompiled with different
options.  (As you'ld need to do to create eg. py27- and py34- versions
of python modules.) Both are things we'd like to have in ports, but they
can be implemented pretty much separately.

Cheers,

Matthew





signature.asc
Description: OpenPGP digital signature

Re: harder and harder to avoid pkg

[Kubilay Kocak]

On 12/10/2016 5:55 PM, Matthew Seaman wrote:
> On 11/10/2016 19:59, Julian Elischer wrote:
>> As the number of dependencies between packages get ever higher, it 
>> becomes more and more difficult to compile packages and the
>> dependence on binary precompiled packages is increased. However
>> binary packages are unsuitable for some situations.  We really need
>> to follow the lead of some of the Linux groups and have -runtime
>> and -devel versions of packages,  OR  we what woudlbe smarter,
>> woudl be to have several "sub manifests" to allow unpacking in
>> different environments.
>> 
>> 
>> A simple example:   libxml2
>> 
>> This package installs include files and libraries and dicumentation
>> etc.
>> 
>> yet if I build an appliance , I want it to only install a singe
>> file.
>> 
>> /usr/local/lib/libxml2.so.2
>> 
>> 
>> The presence of this file will satisfy any runtime dependencies of 
>> packages that require it.
>> 
>> Unfortunately there is no way to install just this file, and still 
>> report that we have the package loaded, so
>> 
>> pkg will always try to reinstall it leading to a huge mess.
>> 
>> My current scheme is to unpack all packages into a larger staging
>> area, and *manually* (scripted) copy out only the files I need, and
>> then copy the pkg database, so that when run on the running
>> appliance, pkg THINKS all the packages are loaded on the appliance,
>> even though only the runtime files are installed. This is what we
>> in the industry call "a hack"  :-) It is also not robust in the
>> face of changing pkg versions.
>> 
>> It would be a lot better it pkg knew it was being asked to install
>> only the runtime set, and coudl accurately  store this information
>> in its database, allowing it to satisfy the needs of other packages
>> that need that dependnency only in a runtime manner.
>> 
>> Is any of this possible at the moment?
>> 
>> suggestions from the ports/pkg community are appreciated..
>> 
>> Julian
> 
> You are describing the 'sub-packages' concept that has been knocking 
> around for some time.  With sub-packages you'ld divide up the result
> of staging each port into various chunks:

Yep, like this:

Mar 6 2016 - https://reviews.freebsd.org/D5563
Ports framework "variants" proof-of-concept (with poudriere support)

Status Report Dec 2015 - Supporting Variants in the Ports Framework

https://www.freebsd.org/news/status/report-2015-10-2015-12.html#Supporting-Variants-in-the-Ports-Framework

> - binaries + config file samples + required data files (the core pkg 
> content) - shlibs - debug symbols - docs - examples - c/c++ headers
> and static or profiling libs (ie. only required for compilation) -
> various additional plugins etc. currently controlled by port options
> 
> Each of these would be packaged separately and can be used
> independently for resolving dependencies.  Building each port would
> result in as many of these sub packages as are applicable.
> 
> Turning OPTIONS into sub-packages will also significantly reduce the 
> number of OPTIONS settings needed in the ports tree (I think bapt had
> an estimate of about a 70% reduction but ICBW) and make the pkg
> system significantly better able to handle more varied user
> requirements without users having to compile their own packages.
> 
> Unfortunately attention has been diverted while there's a lot of
> work going on towards packaging base.  The problem as far as ports
> are concerned is producing several packages out of one port -- it's
> not rocket science level of difficulty to make that change, but the 
> assumption of a one-to-one correspondence between ports and packages
> is deeply rooted, and it's going to take a lot of work to unpick.

Mar 6 2016 - https://reviews.freebsd.org/D5563
Ports framework "variants" proof-of-concept (with poudriere support)

> Happily, the package sets produced for the base system are already 
> divided along these lines, so with a packaged base it is really very 
> easy to produce a stripped down and streamlined base system.
> 
> Cheers,
> 
> Matthew


___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Kubilay Kocak]

On 12/10/2016 5:55 PM, Matthew Seaman wrote:
> On 11/10/2016 19:59, Julian Elischer wrote:
>> As the number of dependencies between packages get ever higher, it 
>> becomes more and more difficult to compile packages and the
>> dependence on binary precompiled packages is increased. However
>> binary packages are unsuitable for some situations.  We really need
>> to follow the lead of some of the Linux groups and have -runtime
>> and -devel versions of packages,  OR  we what woudlbe smarter,
>> woudl be to have several "sub manifests" to allow unpacking in
>> different environments.
>> 
>> 
>> A simple example:   libxml2
>> 
>> This package installs include files and libraries and dicumentation
>> etc.
>> 
>> yet if I build an appliance , I want it to only install a singe
>> file.
>> 
>> /usr/local/lib/libxml2.so.2
>> 
>> 
>> The presence of this file will satisfy any runtime dependencies of 
>> packages that require it.
>> 
>> Unfortunately there is no way to install just this file, and still 
>> report that we have the package loaded, so
>> 
>> pkg will always try to reinstall it leading to a huge mess.
>> 
>> My current scheme is to unpack all packages into a larger staging
>> area, and *manually* (scripted) copy out only the files I need, and
>> then copy the pkg database, so that when run on the running
>> appliance, pkg THINKS all the packages are loaded on the appliance,
>> even though only the runtime files are installed. This is what we
>> in the industry call "a hack"  :-) It is also not robust in the
>> face of changing pkg versions.
>> 
>> It would be a lot better it pkg knew it was being asked to install
>> only the runtime set, and coudl accurately  store this information
>> in its database, allowing it to satisfy the needs of other packages
>> that need that dependnency only in a runtime manner.
>> 
>> Is any of this possible at the moment?
>> 
>> suggestions from the ports/pkg community are appreciated..
>> 
>> Julian
> 
> You are describing the 'sub-packages' concept that has been knocking 
> around for some time.  With sub-packages you'ld divide up the result
> of staging each port into various chunks:

Yep, like this:

Ports framework "variants" proof-of-concept (with poudriere support)
Mar 6 2016 - https://reviews.freebsd.org/D5563

> - binaries + config file samples + required data files (the core pkg 
> content) - shlibs - debug symbols - docs - examples - c/c++ headers
> and static or profiling libs (ie. only required for compilation) -
> various additional plugins etc. currently controlled by port options
> 
> Each of these would be packaged separately and can be used
> independently for resolving dependencies.  Building each port would
> result in as many of these sub packages as are applicable.
> 
> Turning OPTIONS into sub-packages will also significantly reduce the 
> number of OPTIONS settings needed in the ports tree (I think bapt had
> an estimate of about a 70% reduction but ICBW) and make the pkg
> system significantly better able to handle more varied user
> requirements without users having to compile their own packages.
> 
> Unfortunately attention has been diverted while there's a lot of
> work going on towards packaging base.  The problem as far as ports
> are concerned is producing several packages out of one port -- it's
> not rocket science level of difficulty to make that change, but the 
> assumption of a one-to-one correspondence between ports and packages
> is deeply rooted, and it's going to take a lot of work to unpick.

Ports framework "variants" proof-of-concept (with poudriere support)
Mar 6 2016 - https://reviews.freebsd.org/D5563

> Happily, the package sets produced for the base system are already 
> divided along these lines, so with a packaged base it is really very 
> easy to produce a stripped down and streamlined base system.
> 
> Cheers,
> 
> Matthew


___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Julian Elischer]

On 12/10/2016 1:13 AM, Vlad K. wrote:

On 2016-10-11 20:59, Julian Elischer wrote:

are unsuitable for some situations.  We really need to follow the lead
of some of the Linux groups and have -runtime and -devel versions of
packages,  OR  we what woudlbe smarter, woudl be to have several "sub
manifests" to allow unpacking in different environments.


Is as adding a "HEADERS" or whatever you want to call the option to 
ports, a solution? Like we have DOC for documentation, an option 
that could be PLIST sub'd and switch installation of 
include/whatever.h and friends?
what I really need is a RUNTIME option that produces a package with 
only those files needed to satisfy external run-time depdencies, or 
the actual demands of the user itself. However since those files are 
all in the regular package, It'd make sense to just apply the regular 
package to some filter that only allowed those files to be extracted. 
For many packages the whole output would be a single file. (This would 
be true for any package that produces a single .so such as libjpeg or 
libtiff etc. ). The pkg database would however report the package 
being installed, thus satisfying other packages that look in the 
database for dependencies.  Giving it another name (e.g. 
foo-runtime-3.2 ) would make the dependencies not match it.







Yes it's a ton of work requiring to go through many ports, but 
looking at a random sample, it could be scripted and manual labor 
reduced.


To me something like this sounds very much consistent what other 
options, like DOC and MANPAGES, already do. And with individual 
options you don't presume package roles like -dev or -runtime or 
-whatever and you can combine as you want them.


And eventually if, hopefully when, package variants are implemented, 
maybe the official pkg repo can include all the variants, but then, 
I think, that's only a matter of logistics and resource available to 
build all those combinations and store them. But the basic mechanism 
for it should be a port option, imho.






___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Vlad K.]

On 2016-10-11 20:59, Julian Elischer wrote:

are unsuitable for some situations.  We really need to follow the lead
of some of the Linux groups and have -runtime and -devel versions of
packages,  OR  we what woudlbe smarter, woudl be to have several "sub
manifests" to allow unpacking in different environments.


Is as adding a "HEADERS" or whatever you want to call the option to 
ports, a solution? Like we have DOC for documentation, an option that 
could be PLIST sub'd and switch installation of include/whatever.h and 
friends?


Yes it's a ton of work requiring to go through many ports, but looking 
at a random sample, it could be scripted and manual labor reduced.


To me something like this sounds very much consistent what other 
options, like DOC and MANPAGES, already do. And with individual options 
you don't presume package roles like -dev or -runtime or -whatever and 
you can combine as you want them.


And eventually if, hopefully when, package variants are implemented, 
maybe the official pkg repo can include all the variants, but then, I 
think, that's only a matter of logistics and resource available to build 
all those combinations and store them. But the basic mechanism for it 
should be a port option, imho.




--

Vlad K.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Andrea Venturoli]

On 10/12/16 09:24, Matthieu Volat wrote:


And GNU/Linuxes can be a PITA when you have to track -dev(el) packages (which 
sometimes really requires -bin, -app or whatever), or worst, describe to people 
how they are supposed to build your software with weird subpackage names.

I really like that ports provides the software project as intended by upstream 
(modulo options).


Just a "me too" here!

 bye
av.

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Matthieu Volat]

On Tue, 11 Oct 2016 11:59:47 -0700
Julian Elischer  wrote:

> As the number of dependencies between packages get ever higher, it 
> becomes more and more difficult to compile packages and the dependence 
> on binary precompiled packages is increased. However binary packages 
> are unsuitable for some situations.  We really need to follow the lead 
> of some of the Linux groups and have -runtime and -devel versions of 
> packages,  OR  we what woudlbe smarter, woudl be to have several "sub 
> manifests" to allow unpacking in different environments.
> [...]
> 

And GNU/Linuxes can be a PITA when you have to track -dev(el) packages (which 
sometimes really requires -bin, -app or whatever), or worst, describe to people 
how they are supposed to build your software with weird subpackage names.

I really like that ports provides the software project as intended by upstream 
(modulo options).

-- 
Matthieu Volat 


pgpL0u4FDI6tS.pgp
Description: OpenPGP digital signature

Re: harder and harder to avoid pkg

[Matthew Seaman]

On 11/10/2016 19:59, Julian Elischer wrote:
> As the number of dependencies between packages get ever higher, it
> becomes more and more difficult to compile packages and the dependence
> on binary precompiled packages is increased. However binary packages are
> unsuitable for some situations.  We really need to follow the lead of
> some of the Linux groups and have -runtime and -devel versions of
> packages,  OR  we what woudlbe smarter, woudl be to have several "sub
> manifests" to allow unpacking in different environments.
> 
> 
> A simple example:   libxml2
> 
> This package installs include files and libraries and dicumentation etc.
> 
> yet if I build an appliance , I want it to only install a singe file.
> 
> /usr/local/lib/libxml2.so.2
> 
> 
> The presence of this file will satisfy any runtime dependencies of
> packages that require it.
> 
> Unfortunately there is no way to install just this file, and still
> report that we have the package loaded, so
> 
> pkg will always try to reinstall it leading to a huge mess.
> 
> My current scheme is to unpack all packages into a larger staging area,
> and *manually* (scripted) copy out only the files I need, and then copy
> the pkg database, so that when run on the running appliance, pkg THINKS
> all the packages are loaded on the appliance, even though only the
> runtime files are installed. This is what we in the industry call "a
> hack"  :-) It is also not robust in the face of changing pkg versions.
> 
> It would be a lot better it pkg knew it was being asked to install only
> the runtime set, and coudl accurately  store this information in its
> database, allowing it to satisfy the needs of other packages that need
> that dependnency only in a runtime manner.
> 
> Is any of this possible at the moment?
> 
> suggestions from the ports/pkg community are appreciated..
> 
> Julian

You are describing the 'sub-packages' concept that has been knocking
around for some time.  With sub-packages you'ld divide up the result of
staging each port into various chunks:

  - binaries + config file samples + required data files (the core pkg
content)
  - shlibs
  - debug symbols
  - docs
  - examples
  - c/c++ headers and static or profiling libs (ie. only required for
compilation)
  - various additional plugins etc. currently controlled by port options

Each of these would be packaged separately and can be used independently
for resolving dependencies.  Building each port would result in as many
of these sub packages as are applicable.

Turning OPTIONS into sub-packages will also significantly reduce the
number of OPTIONS settings needed in the ports tree (I think bapt had an
estimate of about a 70% reduction but ICBW) and make the pkg system
significantly better able to handle more varied user requirements
without users having to compile their own packages.

Unfortunately attention has been diverted while there's a lot of work
going on towards packaging base.  The problem as far as ports are
concerned is producing several packages out of one port -- it's not
rocket science level of difficulty to make that change, but the
assumption of a one-to-one correspondence between ports and packages is
deeply rooted, and it's going to take a lot of work to unpick.

Happily, the package sets produced for the base system are already
divided along these lines, so with a packaged base it is really very
easy to produce a stripped down and streamlined base system.

Cheers,

Matthew



signature.asc
Description: OpenPGP digital signature

Re: harder and harder to avoid pkg

[Alfred Perlstein]

I feel like creative use of run/build-depends would work but I'm a bit tired 
now. Well you probably don't want any or near zero deps down a library depends 
path. 

Sent from my iPhone

> On Oct 11, 2016, at 10:08 PM, Julian Elischer  wrote:
> 
>> On 11/10/2016 5:34 PM, Alfred Perlstein wrote:
>> Make a slave port with an abbreviated pkg-plist bruh. ;)
> yeeess, good idea, but that won't satisfy the dependency requirements of 
> other packages... you need to fool other packages, and that's the hard part. 
> The way to do this is I think for pkg to have the ability to have two 
> manifests.
> 
> We are doing similar to what Roger says, but it's just so much work...
> 
>> 
>> -Alfred
>> 
>> 
>>> On 10/11/16 11:59 AM, Julian Elischer wrote:
>>> As the number of dependencies between packages get ever higher, it becomes 
>>> more and more difficult to compile packages and the dependence on binary 
>>> precompiled packages is increased. However binary packages are unsuitable 
>>> for some situations.  We really need to follow the lead of some of the 
>>> Linux groups and have -runtime and -devel versions of packages, OR  we what 
>>> woudlbe smarter, woudl be to have several "sub manifests" to allow 
>>> unpacking in different environments.
>>> 
>>> 
>>> A simple example:   libxml2
>>> 
>>> This package installs include files and libraries and dicumentation etc.
>>> 
>>> yet if I build an appliance , I want it to only install a singe file.
>>> 
>>> /usr/local/lib/libxml2.so.2
>>> 
>>> 
>>> The presence of this file will satisfy any runtime dependencies of packages 
>>> that require it.
>>> 
>>> Unfortunately there is no way to install just this file, and still report 
>>> that we have the package loaded, so
>>> 
>>> pkg will always try to reinstall it leading to a huge mess.
>>> 
>>> My current scheme is to unpack all packages into a larger staging area, and 
>>> *manually* (scripted) copy out only the files I need, and then copy the pkg 
>>> database, so that when run on the running appliance, pkg THINKS all the 
>>> packages are loaded on the appliance, even though only the runtime files 
>>> are installed. This is what we in the industry call "a hack"  :-) It is 
>>> also not robust in the face of changing pkg versions.
>>> 
>>> It would be a lot better it pkg knew it was being asked to install only the 
>>> runtime set, and coudl accurately  store this information in its database, 
>>> allowing it to satisfy the needs of other packages that need that 
>>> dependnency only in a runtime manner.
>>> 
>>> Is any of this possible at the moment?
>>> 
>>> suggestions from the ports/pkg community are appreciated..
>>> 
>>> Julian
>>> 
>>> 
>>> ___
>>> freebsd-ports@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
>>> To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
>>> 
>> 
>> 
> 

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Julian Elischer]

On 11/10/2016 5:34 PM, Alfred Perlstein wrote:

Make a slave port with an abbreviated pkg-plist bruh. ;)
yeeess, good idea, but that won't satisfy the dependency requirements 
of other packages... you need to fool other packages, and that's the 
hard part. The way to do this is I think for pkg to have the ability 
to have two manifests.


We are doing similar to what Roger says, but it's just so much work...



-Alfred


On 10/11/16 11:59 AM, Julian Elischer wrote:
As the number of dependencies between packages get ever higher, it 
becomes more and more difficult to compile packages and the 
dependence on binary precompiled packages is increased. However 
binary packages are unsuitable for some situations.  We really need 
to follow the lead of some of the Linux groups and have -runtime 
and -devel versions of packages, OR  we what woudlbe smarter, woudl 
be to have several "sub manifests" to allow unpacking in different 
environments.



A simple example:   libxml2

This package installs include files and libraries and dicumentation 
etc.


yet if I build an appliance , I want it to only install a singe file.

/usr/local/lib/libxml2.so.2


The presence of this file will satisfy any runtime dependencies of 
packages that require it.


Unfortunately there is no way to install just this file, and still 
report that we have the package loaded, so


pkg will always try to reinstall it leading to a huge mess.

My current scheme is to unpack all packages into a larger staging 
area, and *manually* (scripted) copy out only the files I need, and 
then copy the pkg database, so that when run on the running 
appliance, pkg THINKS all the packages are loaded on the appliance, 
even though only the runtime files are installed. This is what we 
in the industry call "a hack"  :-) It is also not robust in the 
face of changing pkg versions.


It would be a lot better it pkg knew it was being asked to install 
only the runtime set, and coudl accurately  store this information 
in its database, allowing it to satisfy the needs of other packages 
that need that dependnency only in a runtime manner.


Is any of this possible at the moment?

suggestions from the ports/pkg community are appreciated..

Julian


___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to 
"freebsd-ports-unsubscr...@freebsd.org"







___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Alfred Perlstein]

Make a slave port with an abbreviated pkg-plist bruh.  ;)

-Alfred


On 10/11/16 11:59 AM, Julian Elischer wrote:
As the number of dependencies between packages get ever higher, it 
becomes more and more difficult to compile packages and the dependence 
on binary precompiled packages is increased. However binary packages 
are unsuitable for some situations.  We really need to follow the lead 
of some of the Linux groups and have -runtime and -devel versions of 
packages, OR  we what woudlbe smarter, woudl be to have several "sub 
manifests" to allow unpacking in different environments.



A simple example:   libxml2

This package installs include files and libraries and dicumentation etc.

yet if I build an appliance , I want it to only install a singe file.

/usr/local/lib/libxml2.so.2


The presence of this file will satisfy any runtime dependencies of 
packages that require it.


Unfortunately there is no way to install just this file, and still 
report that we have the package loaded, so


pkg will always try to reinstall it leading to a huge mess.

My current scheme is to unpack all packages into a larger staging 
area, and *manually* (scripted) copy out only the files I need, and 
then copy the pkg database, so that when run on the running appliance, 
pkg THINKS all the packages are loaded on the appliance, even though 
only the runtime files are installed. This is what we in the industry 
call "a hack"  :-) It is also not robust in the face of changing pkg 
versions.


It would be a lot better it pkg knew it was being asked to install 
only the runtime set, and coudl accurately  store this information in 
its database, allowing it to satisfy the needs of other packages that 
need that dependnency only in a runtime manner.


Is any of this possible at the moment?

suggestions from the ports/pkg community are appreciated..

Julian


___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"



___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Julian Elischer]

On 11/10/2016 12:51 PM, olli hauer wrote:

On 2016-10-11 20:59, Julian Elischer wrote:

As the number of dependencies between packages get ever higher, it becomes more and more 
difficult to compile packages and the dependence on binary precompiled packages is 
increased. However binary packages are unsuitable for some situations.  We really need to 
follow the lead of some of the Linux groups and have -runtime and -devel versions of 
packages,  OR  we what woudlbe smarter, woudl be to have several "sub 
manifests" to allow unpacking in different environments.


A simple example:   libxml2

This package installs include files and libraries and dicumentation etc.

yet if I build an appliance , I want it to only install a singe file.

/usr/local/lib/libxml2.so.2


The presence of this file will satisfy any runtime dependencies of packages 
that require it.

Unfortunately there is no way to install just this file, and still report that 
we have the package loaded, so

pkg will always try to reinstall it leading to a huge mess.

My current scheme is to unpack all packages into a larger staging area, and *manually* 
(scripted) copy out only the files I need, and then copy the pkg database, so that when 
run on the running appliance, pkg THINKS all the packages are loaded on the appliance, 
even though only the runtime files are installed. This is what we in the industry call 
"a hack"  :-) It is also not robust in the face of changing pkg versions.

It would be a lot better it pkg knew it was being asked to install only the 
runtime set, and coudl accurately  store this information in its database, 
allowing it to satisfy the needs of other packages that need that dependnency 
only in a runtime manner.

Is any of this possible at the moment?

suggestions from the ports/pkg community are appreciated..

Julian


Hm, do you build your own packages or using pre build packages?

yes we build them, for simple ones, but
for java or some of the more complicated ones, we PREFER to use the 
precompiled ones.
This is because (as mentioned in the email) the explosion of 
dependencies means that compiling our own is getting less and less 
feasible.

Especially if we have our own changes in some of the prerequisite modules.



just for interest and testing the following hack will do what you want, but 
results in duplicate category and some other minor errors (needs some better 
hacking ...)

$ cat libxml2/Makefile.local
post-stage:
@${RM} -rf ${STAGEDIR}${PREFIX}/include/libxml2
@${ECHO} 'lib/libxml2.so.2' > ${TMPPLIST}
That's true and in fact we have the technology in house to do that, 
but it's a very messy solution, and not terribly reproducible, 
requiring lots of human intervention in the long run.  it also 
produces accounting issues as now we need to have two versions of each 
pkg (with the same name),
One that we need to install into the build environment (with include 
files, etc) and one for installing into the target appliance image.





A better method would be to use a tool like portsharker together with 
ports-mgmt/poudriere(-devel)
A really short hint how to use it can be found here:
  https://github.com/freebsd/poudriere/blob/release-3.1/doc/portshaker.wiki

Anyway, until now should be possible with some effort, perhaps it is possible 
to get for such purpose an additional make target that is running after stage 
but before the package is created.




___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[Mathieu Arnold]

Le 11/10/2016 à 20:59, Julian Elischer a écrit :
> As the number of dependencies between packages get ever higher, it
> becomes more and more difficult to compile packages and the dependence
> on binary precompiled packages is increased. However binary packages
> are unsuitable for some situations.  We really need to follow the lead
> of some of the Linux groups and have -runtime and -devel versions of
> packages,  OR  we what woudlbe smarter, woudl be to have several "sub
> manifests" to allow unpacking in different environments.

That would be great, I can't wait to see the patches.


-- 
Mathieu Arnold




signature.asc
Description: OpenPGP digital signature

Re: harder and harder to avoid pkg

[Roger Marquis]

On Tue, 11 Oct 2016, Julian Elischer wrote:
*manually* (scripted) copy out only the files I need, and then copy the pkg 
database, so that when run on the running appliance, pkg THINKS all the 
packages are loaded


We do something similar using prebuilt (pkg fetch) or locally built
(make package) packages by:

 1) untarring the .pkg:
cd /tmp && tar xzvf {/usr/ports/packages/All,/var/cache/pkg}/...

 2) unpacking the manifests:
   cat -- +COMPACT_MANIFEST | jq -a 'del(.shlibs_required,.deps)' > \
   COMPACT_MANIFEST.tmp
   cat -- +MANIFEST | jq '.' > MANIFEST.tmp

 3) removing unecessary files and dependencies:
vi *MANIFEST.tmp
  :s/DOCS": "on/DOCS": "off/
  :g,share/doc,d
  ...
rm -rf ./usr/local/{same files and deps as vi}

 4) and repacking into a new, minimal pkg:
mv COMPACT_MANIFEST.tmp +COMPACT_MANIFEST && \
mv MANIFEST.tmp +MANIFEST && \
tar cf - ... | (cd / && tar xfBp -) && \
pkg create -M +MANIFEST

This tends to be easier than patching port/{Makefile,pkg-plist,files/...},
keeps /var/db/pkg/local.sqlite valid and works well with 'pkg audit'.
Should also, theoretically, be easy enough to roll into a metaport.

Thanks to Devin for the original idea.

YMMV,
Roger
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: harder and harder to avoid pkg

[olli hauer]

On 2016-10-11 20:59, Julian Elischer wrote:
> As the number of dependencies between packages get ever higher, it becomes 
> more and more difficult to compile packages and the dependence on binary 
> precompiled packages is increased. However binary packages are unsuitable for 
> some situations.  We really need to follow the lead of some of the Linux 
> groups and have -runtime and -devel versions of packages,  OR  we what 
> woudlbe smarter, woudl be to have several "sub manifests" to allow unpacking 
> in different environments.
> 
> 
> A simple example:   libxml2
> 
> This package installs include files and libraries and dicumentation etc.
> 
> yet if I build an appliance , I want it to only install a singe file.
> 
> /usr/local/lib/libxml2.so.2
> 
> 
> The presence of this file will satisfy any runtime dependencies of packages 
> that require it.
> 
> Unfortunately there is no way to install just this file, and still report 
> that we have the package loaded, so
> 
> pkg will always try to reinstall it leading to a huge mess.
> 
> My current scheme is to unpack all packages into a larger staging area, and 
> *manually* (scripted) copy out only the files I need, and then copy the pkg 
> database, so that when run on the running appliance, pkg THINKS all the 
> packages are loaded on the appliance, even though only the runtime files are 
> installed. This is what we in the industry call "a hack"  :-) It is also not 
> robust in the face of changing pkg versions.
> 
> It would be a lot better it pkg knew it was being asked to install only the 
> runtime set, and coudl accurately  store this information in its database, 
> allowing it to satisfy the needs of other packages that need that dependnency 
> only in a runtime manner.
> 
> Is any of this possible at the moment?
> 
> suggestions from the ports/pkg community are appreciated..
> 
> Julian
> 

Hm, do you build your own packages or using pre build packages?

just for interest and testing the following hack will do what you want, but 
results in duplicate category and some other minor errors (needs some better 
hacking ...)

$ cat libxml2/Makefile.local
post-stage:
@${RM} -rf ${STAGEDIR}${PREFIX}/include/libxml2
@${ECHO} 'lib/libxml2.so.2' > ${TMPPLIST}


A better method would be to use a tool like portsharker together with 
ports-mgmt/poudriere(-devel)
A really short hint how to use it can be found here:
 https://github.com/freebsd/poudriere/blob/release-3.1/doc/portshaker.wiki

Anyway, until now should be possible with some effort, perhaps it is possible 
to get for such purpose an additional make target that is running after stage 
but before the package is created.

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

harder and harder to avoid pkg

[Julian Elischer]

As the number of dependencies between packages get ever higher, it 
becomes more and more difficult to compile packages and the dependence 
on binary precompiled packages is increased. However binary packages 
are unsuitable for some situations.  We really need to follow the lead 
of some of the Linux groups and have -runtime and -devel versions of 
packages,  OR  we what woudlbe smarter, woudl be to have several "sub 
manifests" to allow unpacking in different environments.



A simple example:   libxml2

This package installs include files and libraries and dicumentation etc.

yet if I build an appliance , I want it to only install a singe file.

/usr/local/lib/libxml2.so.2


The presence of this file will satisfy any runtime dependencies of 
packages that require it.


Unfortunately there is no way to install just this file, and still 
report that we have the package loaded, so


pkg will always try to reinstall it leading to a huge mess.

My current scheme is to unpack all packages into a larger staging 
area, and *manually* (scripted) copy out only the files I need, and 
then copy the pkg database, so that when run on the running appliance, 
pkg THINKS all the packages are loaded on the appliance, even though 
only the runtime files are installed. This is what we in the industry 
call "a hack"  :-) It is also not robust in the face of changing pkg 
versions.


It would be a lot better it pkg knew it was being asked to install 
only the runtime set, and coudl accurately  store this information in 
its database, allowing it to satisfy the needs of other packages that 
need that dependnency only in a runtime manner.


Is any of this possible at the moment?

suggestions from the ports/pkg community are appreciated..

Julian


___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"