Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-30 Thread Bertrand Jacquin
D'ar yaou 28 a viz Genver 2010 e 23 eur 31, « Robin H. Johnson » he deus 
skrivet :
 On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
  So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
  names function of USE, like : 
CHOST=x86_64-pc-linux-gnu
ACCEPT_KEYWORDS=amd64
tcpdump
  
USE=+chroot +ipv6 -ssl -test -samba -smi
is placed under 
  /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7b4416738.tbz2
  
USE=+chroot +ipv6 +ssl -test +samba -smi
is placed under 
  /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb495980bd.tbz2
 A word of warning on this.
 
 Simply capturing the USE flags is not enough, the libraries that a
 binary is linked against can cause problems as well, as the dependency
 imposed by linking is much stricter than the RDEPEND in the ebuild.
 
 tcpdump for example has:
 RDEPEND=... ssl? ( =dev-libs/openssl-0.9.6m )

Of course, but real *DEPEND could be added in the xpak part of tbz2 and
resolv depedencies by the way.

 But if you built and linked against openssl-0.9.7*, then upgraded to
 openssl-0.9.8, you can expect breakage.
 
 Similar things happened with binaries linked against libmysqlclient in
 the past.
 
 As to my own take on a solution to the problem for work stuff, we simply
 maintain one chroot per unique environment (we only have 5 envs), and
 build in the chroot, then use those binpkgs on the boxes.

I did that already, but it's a bit heavy to maintain and do much
repetitive (maybe it's not the good word) as many packages would be
build with the same USE (in the case of same arch, chost etc).

-- 
Beber


pgppYr2q0aw1b.pgp
Description: PGP signature


Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-30 Thread Bertrand Jacquin
D'ar gwener 29 a viz Genver 2010 e 00 eur 23, « Philipp Riegger » he deus 
skrivet :
 On Thu, 28 Jan 2010 16:17:41 +0100
 Beber be...@meleeweb.net wrote:
 
  So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
  names function of USE, like : 
CHOST=x86_64-pc-linux-gnu
ACCEPT_KEYWORDS=amd64
tcpdump
  
USE=+chroot +ipv6 -ssl -test -samba -smi
is placed
  under 
  /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7b4416738.tbz2
  
USE=+chroot +ipv6 +ssl -test +samba -smi
is placed
  under 
  /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb495980bd.tbz2
 
 So these are the SHA1 sums of the packages? Nice.

No, the SHA1 is USE var SHA1.

sha1sum  $USE

-- 
Beber


pgpBKVzwhXru8.pgp
Description: PGP signature


Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-30 Thread Zac Medico
On 01/28/2010 09:24 PM, Max Arnold wrote:
 On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
 So, do you guys plan to implement a such thing ? That's one of the
 features that is mostly missing imho. The principal miss in on client
 side as I have tools to manage packages but would like to not have too
 much specific scripts on client side.
 
 I like the way it done in OpenEmbedded. You have the tree of recipes (think 
 of portage tree)
 and bunch of targets. For each target BitBake can generate binary release and 
 package feed.
 Client package management is lightweight and does not require BitBake, 
 recipes tree and even
 python. At least this is my lame interpretation of how it works :)

You can do something similar using the emerge --config-root option.
You'd just use a different --config-root for each target, and each
of those would have a separate $PKGDIR.

 Maybe this metadistribution approach is cleaner than binary package support 
 in emerge. If
 user wants to compile packages on the client, he uses portage. If not - he 
 can setup build
 server for multiple targets and completely drop portage from client machines. 
 The only thing
 client should know is feed url with full list of binary packages. And I do 
 not think client
 should deal with USE flags - for large installations unification is the only 
 sane way to scale.

The clients need sys-apps/portage installed, but not the whole
portage tree (although the profiles directory can be useful for
the profile and package moves). The clients should set
PORTAGE_BINHOST in make.conf, so that binary packages are
automatically downloaded with the emerge -g option.
-- 
Thanks,
Zac



Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-30 Thread Max Arnold
On Sat, Jan 30, 2010 at 03:59:27AM -0800, Zac Medico wrote:
 On 01/28/2010 09:24 PM, Max Arnold wrote:
  On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
  So, do you guys plan to implement a such thing ? That's one of the
  features that is mostly missing imho. The principal miss in on client
  side as I have tools to manage packages but would like to not have too
  much specific scripts on client side.
  
  I like the way it done in OpenEmbedded. You have the tree of recipes (think 
  of portage tree)
  and bunch of targets. For each target BitBake can generate binary release 
  and package feed.
  Client package management is lightweight and does not require BitBake, 
  recipes tree and even
  python. At least this is my lame interpretation of how it works :)
 
 You can do something similar using the emerge --config-root option.
 You'd just use a different --config-root for each target, and each
 of those would have a separate $PKGDIR.

Can someone share his experience of using Catalyst or Metro as generator of 
binary package
updates for precompiled client machines?  How smooth it is in the long run?



Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-30 Thread Jacob Godserv
On Sat, Jan 30, 2010 at 10:13, Max Arnold lwa...@gmail.com wrote:
 Can someone share his experience of using Catalyst or Metro as generator of 
 binary package
 updates for precompiled client machines?  How smooth it is in the long run?

I use metro (wrote a little HOWTO on it: 
http://neuvoo.org/wiki/index.php?title=Create_Image/Building ) and it
asks emerge to --buildpkg at every stage. It keeps binaries separate
per stage. The end result, as long as you told it not to clean, is
three folders with a binary repo in each. It will re-use these
binaries in future stage builds, if you don't clean it out.

If you don't want it building stages, it's very easy to configure
metro to do exactly what you want. It still has to be figured out to
some extent, but the format is much simpler than catalyst and takes
less time to customize properly.

I'm not sure what long run looks like as I have yet to enter the
long run. As far as I can tell, metro will always build and update
binary repositories when you ask it to. ;)

-- 
Jacob

For then there will be great distress, unequaled
from the beginning of the world until now — and never
to be equaled again. If those days had not been cut
short, no one would survive, but for the sake of the
elect those days will be shortened.

Are you ready?



Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-29 Thread Antoni Grzymala
Max Arnold dixit (2010-01-29, 12:24):

 On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
  So, do you guys plan to implement a such thing ? That's one of the
  features that is mostly missing imho. The principal miss in on
  client side as I have tools to manage packages but would like to not
  have too much specific scripts on client side.
 
 I like the way it done in OpenEmbedded. You have the tree of recipes
 (think of portage tree) and bunch of targets. For each target BitBake
 can generate binary release and package feed. Client package
 management is lightweight and does not require BitBake, recipes tree
 and even python. At least this is my lame interpretation of how it
 works :)
 
 Maybe this metadistribution approach is cleaner than binary package
 support in emerge. If user wants to compile packages on the client, he
 uses portage. If not - he can setup build server for multiple targets
 and completely drop portage from client machines. The only thing
 client should know is feed url with full list of binary packages. And
 I do not think client should deal with USE flags - for large
 installations unification is the only sane way to scale.

I think something similar is very nicely done on R-Path based
distributions (with flavours [1]). Conary itself also seems to be a
pretty well thought out package manager.

[1] http://docs.rpath.com/conary/Conaryopedia/sect-flavors.html

-- 
[a]



Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-29 Thread Philipp Riegger
On Thu, 28 Jan 2010 22:31:04 +
Robin H. Johnson robb...@gentoo.org wrote:

 But if you built and linked against openssl-0.9.7*, then upgraded to
 openssl-0.9.8, you can expect breakage.

But can't this be kind of ignored and solves the Gentoo way? If the
binpkg does not work, don't install it and simply rebuild it (if not
stated otherwise on the command line)?

Philipp



Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-29 Thread Robin H. Johnson
On Fri, Jan 29, 2010 at 11:52:40AM +0100, Philipp Riegger wrote:
 On Thu, 28 Jan 2010 22:31:04 +
 Robin H. Johnson robb...@gentoo.org wrote:
  But if you built and linked against openssl-0.9.7*, then upgraded to
  openssl-0.9.8, you can expect breakage.
 But can't this be kind of ignored and solves the Gentoo way? If the
 binpkg does not work, don't install it and simply rebuild it (if not
 stated otherwise on the command line)?
The gentoo way in this case is revdep-rebuild, and you've already
stated that your servers do not have a compiler.

You need to either expose the linkage dependencies outside the binpkg,
or introspect it when you're about to install it to find that it's NOT
going to work.

zmedico/portage folk: Can we have a phase during binpkg install, that
lets us check ELF linkages before the files get moving to ${ROOT}?

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee  Infrastructure Lead
E-Mail : robb...@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85


pgpLTkndg3Q5T.pgp
Description: PGP signature


Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-29 Thread Zac Medico
On 01/29/2010 01:44 PM, Robin H. Johnson wrote:
 zmedico/portage folk: Can we have a phase during binpkg install, that
 lets us check ELF linkages before the files get moving to ${ROOT}?

Yeah, portage-2.2 has a LinkageMap class that we can use to verify
that all the needed sonames are present.
-- 
Thanks,
Zac



[gentoo-dev] Building custom package for multi-arch/system

2010-01-28 Thread Beber
Hi,

This mail is continuation of a discussion last month on IRC on
#gentoo-portage. As I think it's larger than just portage array, here it
is.

I have one server dedicated for compilation (xm-compile), and many
applicative servers (xmred5, lemonhead, etc).

Currently I use the --buildpkg on xm-compile to build packages
emergeable by --getbinpkgonly. Everything is fine except that I'm not
able to --getbinpkg function of USE flags, CFLAGS, CHOST nor ACCEPT_KEYWORDS.

So all packages on all servers are build in the same way with same USE.
But I would like to avoid it. I have mx servers (exim) that need mysql
lib, so mysql USE is there. I have mysql servers so -minimal USE is
there. I don't want to install a full mysql on mx servers, I just need
mysql lib.

So I'd like to build custom packages for almost everything.

I don't want to build packages on servers for security reasons (somes
could reproach me that, but it's not the topic ;) and for a limitation
of disk usage.

So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
names function of USE, like : 
  CHOST=x86_64-pc-linux-gnu
  ACCEPT_KEYWORDS=amd64
  tcpdump

  USE=+chroot +ipv6 -ssl -test -samba -smi
  is placed under 
/data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7b4416738.tbz2

  USE=+chroot +ipv6 +ssl -test +samba -smi
  is placed under 
/data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb495980bd.tbz2

Like this, I can easily add on client side a portage bashrc to get the file
and place it in the good place to don't have portage to download it as
it would find the right file with good USE. This implies I need to
emerge 1 package per 1 package which is horrible after some times.

So, do you guys plan to implement a such thing ? That's one of the
features that is mostly missing imho. The principal miss in on client
side as I have tools to manage packages but would like to not have too
much specific scripts on client side.

Thanks,

-- 
Beber


signature.asc
Description: Digital signature


Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-28 Thread Jacob Godserv
(I'm not an official developer.)

On Thu, Jan 28, 2010 at 10:17, Beber be...@meleeweb.net wrote:
 So, do you guys plan to implement a such thing ? That's one of the
 features that is mostly missing imho. The principal miss in on client
 side as I have tools to manage packages but would like to not have too
 much specific scripts on client side.

This is also high on my things to hack list. (Not that I'd ever come
up with something acceptable, but it's worth a try. :)

I would love to see the whole PKGDIR variable vanish altogether and
merge binary builds into ebuilds, creating a sort of apt-get / portage
mixture, all with a standard ebuild setup. By allowing binaries to
take on all the functionality of ebuilds, features like USE flags,
multiple versions per repository, varying CFLAGs, and other such
features are a given. The binary can be downloaded via standard ebuild
methods, even.

Something to chew on.

-- 

For then there will be great distress, unequaled
from the beginning of the world until now — and never
to be equaled again. If those days had not been cut
short, no one would survive, but for the sake of the
elect those days will be shortened.

Are you ready?



Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-28 Thread Gilles Dartiguelongue
Hi,

I have a somewhat similar need and to satisfy it, I'm building packages
in a chroot that has all the common packages then one chroot per
variation. It's probably nowhere scalable enough for you though and
would most likely but much more efficient if package building could be
wrapped so that it builds the package for one setup, then reconfigures
the package and build it for the new setup without cleaning everything
in the middle. Some scripting could probably do that but then I think
this joins well with multiple ABIs discussions as it basically means
portage needs to be able to build the same package several times with
different configuration.

-- 
Gilles Dartiguelongue e...@gentoo.org
Gentoo


signature.asc
Description: Ceci est une partie de message numériquement signée


Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-28 Thread Robin H. Johnson
On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
 So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
 names function of USE, like : 
   CHOST=x86_64-pc-linux-gnu
   ACCEPT_KEYWORDS=amd64
   tcpdump
 
   USE=+chroot +ipv6 -ssl -test -samba -smi
   is placed under 
 /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7b4416738.tbz2
 
   USE=+chroot +ipv6 +ssl -test +samba -smi
   is placed under 
 /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb495980bd.tbz2
A word of warning on this.

Simply capturing the USE flags is not enough, the libraries that a
binary is linked against can cause problems as well, as the dependency
imposed by linking is much stricter than the RDEPEND in the ebuild.

tcpdump for example has:
RDEPEND=... ssl? ( =dev-libs/openssl-0.9.6m )

But if you built and linked against openssl-0.9.7*, then upgraded to
openssl-0.9.8, you can expect breakage.

Similar things happened with binaries linked against libmysqlclient in
the past.

As to my own take on a solution to the problem for work stuff, we simply
maintain one chroot per unique environment (we only have 5 envs), and
build in the chroot, then use those binpkgs on the boxes.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee  Infrastructure Lead
E-Mail : robb...@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85


pgp8j11icauBj.pgp
Description: PGP signature


Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-28 Thread Philipp Riegger
On Thu, 28 Jan 2010 16:17:41 +0100
Beber be...@meleeweb.net wrote:

 So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
 names function of USE, like : 
   CHOST=x86_64-pc-linux-gnu
   ACCEPT_KEYWORDS=amd64
   tcpdump
 
   USE=+chroot +ipv6 -ssl -test -samba -smi
   is placed
 under 
 /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7b4416738.tbz2
 
   USE=+chroot +ipv6 +ssl -test +samba -smi
   is placed
 under 
 /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb495980bd.tbz2
 

So these are the SHA1 sums of the packages? Nice.

I would like to see something like this:

With different CFLAGS, CHOST etc. the user should set a seperate PKGDIR.

USE-flags are encoded in the filename:

Sort USE-flags alphabetically, use a 1 for every enabled and a 0 for
every disabled USE-flag, make this array smaller using hex code ore
base64.

   USE=+chroot +ipv6 -ssl -test -samba -smi

11

tcpdump-4.0.1_pre20090709+c0.tbz2

   USE=+chroot +ipv6 +ssl -test +samba -smi

111010

tcpdump-4.0.1_pre20090709+e2.tbz2

Problem with this is, if anything with the USEflags as we see it from
outside changes (USE-flags are added, deletes, renamed and therefore
change position), this gets none-unique, which can be fixed in several
ways.

Philipp



Re: [gentoo-dev] Building custom package for multi-arch/system

2010-01-28 Thread Max Arnold
On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
 So, do you guys plan to implement a such thing ? That's one of the
 features that is mostly missing imho. The principal miss in on client
 side as I have tools to manage packages but would like to not have too
 much specific scripts on client side.

I like the way it done in OpenEmbedded. You have the tree of recipes (think of 
portage tree)
and bunch of targets. For each target BitBake can generate binary release and 
package feed.
Client package management is lightweight and does not require BitBake, recipes 
tree and even
python. At least this is my lame interpretation of how it works :)

Maybe this metadistribution approach is cleaner than binary package support 
in emerge. If
user wants to compile packages on the client, he uses portage. If not - he can 
setup build
server for multiple targets and completely drop portage from client machines. 
The only thing
client should know is feed url with full list of binary packages. And I do not 
think client
should deal with USE flags - for large installations unification is the only 
sane way to scale.