Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-16 Thread Lars Engels
On Mon, Jul 15, 2013 at 11:52:26AM -0500, Mark Felder wrote:
 On Mon, 15 Jul 2013 11:48:18 -0500, Teske, Devin
 devin.te...@fisglobal.com wrote:
  I think Mark was saying that libfetch doesn't yet support http proxy.
 
 Now that I've googled, I was referring to this PR
 http://www.freebsd.org/cgi/query-pr.cgi?pr=180253
 
 I thought it was complete missing functionality, not just that bug.

libfetch supports proxies even with authentication using HTTP_PROXY_AUTH
environment variable


pgpwNwIYXocDX.pgp
Description: PGP signature


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Baptiste Daroussin
On Sun, Jul 14, 2013 at 10:12:19AM +, Teske, Devin wrote:
 
 On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote:
 
  On 14/07/2013 06:48, Teske, Devin wrote:
  Question: Where can I learn more about the actual format of what's in
  the new tarballs? This is going to be important not for bsdconfig,
  but $work (we have our own build platform; I'm going to have to
  rewrite it from mastering PLIST files to mastering YAML MANIFEST
  files and I want to know all the gritty details).
  
  We do need a pkg-manifest(5) man page, which can double as a
  pkg-tarball(5) page since the manifest file is where most of the
  interesting bits are.
  
  A pkg tarball is a compressed tar archive like so:
  
  lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz
  -rw-r--r--  0 root   wheel 530 Jan  1  1970 +COMPACT_MANIFEST
  -rw-r--r--  0 root   wheel6385 Jan  1  1970 +MANIFEST
  -rw-r--r--  0 root   wheel   17567 Jan  1  1970 +MTREE_DIRS
  -r--r--r--  0 root   wheel   19453 Jul  7 12:26
  /usr/local/etc/bash_completion.d/_pkg.bash
  -r-xr-xr-x  0 root   wheel 629 Jul  7 12:26
  /usr/local/etc/periodic/daily/400.status-pkg
  -r-xr-xr-x  0 root   wheel 823 Jul  7 12:26
  /usr/local/etc/periodic/daily/411.pkg-backup
  -r-xr-xr-x  0 root   wheel 678 Jul  7 12:26
  /usr/local/etc/periodic/daily/490.status-pkg-changes
  -r-xr-xr-x  0 root   wheel2558 Jul  7 12:26
  /usr/local/etc/periodic/security/410.pkg-audit
  -r-xr-xr-x  0 root   wheel 611 Jul  7 12:26
  /usr/local/etc/periodic/security/460.pkg-checksum
  -r--r--r--  0 root   wheel 839 Jul  7 12:26
  /usr/local/etc/pkg.conf.sample
  -r--r--r--  0 root   wheel   43432 Jul  7 12:26 /usr/local/include/pkg.h
  -r--r--r--  0 root   wheel  727558 Jul  7 12:26 /usr/local/lib/libpkg.a
  lrwxr-xr-x  0 root   wheel   0 Jul  7 12:26 /usr/local/lib/libpkg.so
  - libpkg.so.1
  -r--r--r--  0 root   wheel 1227064 Jul  7 12:26 /usr/local/lib/libpkg.so.1
  -rw-r--r--  0 root   wheel 187 Jul  7 12:26
  /usr/local/libdata/pkgconfig/pkg.pc
  [... etc ...]
  
 
 Interesting. I notice that (while looking ahead to see a prefix: of 
 /usr/local in the +MANIFEST), the tarball itself has files that include 
 /usr/local in their path.
 
 Does pkg handle packages where the prefix (in +MANIFEST) is /usr/local 
 _but_ tar tf of the unxz'd tarball is _not_ /usr/local? (e.g. pkg_install 
 style) or is this the new way going forward?
 
 
 
  There must at least be a +MANIFEST -- other meta data files are
  optional.  +COMPACT_MANIFEST is a subset of the full +MANIFEST.  They're
  both YAML documents.  +COMPACT_MANIFEST looks like this, for example:
  
  ---
  name: pkg
  version: 1.1.4
  origin: ports-mgmt/pkg
  comment: New generation package manager
  arch: freebsd:9:x86:64
  www: 
  https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=LPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0As=30b27cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d
  maintainer: port...@freebsd.org
  prefix: /usr/local
  licenselogic: single
  licenses:
  - BSD
  flatsize: 6311507
  desc: |
   New Generation package management tool for FreeBSD
  
   WWW: http://wiki.freebsd.org/pkgng
  categories:
  - ports-mgmt
  shlibs_required:
  - libpkg.so.1
  shlibs_provided:
  - libpkg.so.1
  message: |
   If you are upgrading from the old package format, first run:
  
 # pkg2ng
  
 
 Nice. Very nice.
 
 
 
  +MTREE_DIRS is a compatibility thing with the old pkg_tools.  It's not
  needed in general as +MANIFEST can provide all that meta data itself.
  It isn't going to be deprecated for at least as long as the ports tree
  continues to support pkg_tools though.
  
  Beyond that, the rest of the pkg tarball just contains a tar archive of
  all the files, directories, sym-links etc to be installed by the
  package.  Note that pkg(8) has no problem with creating an empty
  directory for a package, unlike pkg_tools.
  
 
 Excellent!
 
 
  Now, while you can grovel through the details of pkg tarballs and
  internal data formats like this, be aware: the format of the manifest
  and the details of the meta-data included in the pkg-tarballs is subject
  to change without warning as we develop pkg(8) further.  We only promise
  API stability through the pkg(8) commands or for the libpkg.so library
  functions; reports of breakage from usage outside those APIs will
  receive little sympathy.
  
 
 Understood.
 
 So just to give you a better idea of how we manage packages here.
 
 We've realized that if you want to package for FreeBSD (in 9 and older), you 
 could get by alright if you knew how to create a +CONTENTS file from scratch. 
 For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an awful lot 
 like a SPECFILE (they are both in-fact YAML).

The only thing you need if you really want to package is a small subset of the
manifest written in YAML:

Have a look at what the ports 

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Baptiste Daroussin
On Sun, Jul 14, 2013 at 07:29:50AM +, Teske, Devin wrote:
 
   Or you
  can maintain the old cruft for your business -- just don't expect
  anyone else to use it, or even want to.
  
 
 
 I have no intention of making old-world packages... but I also have no 
 intention of using pkg create.

So you are taking as an exemple rpm but with RPM you do want to use what they do
provide, aka rpmbuild but with pkgng you want to manually handwrite the packages
because we haven't hidden the format behind opaque names and format.

You know in that case you should probably create your own RPM without RPM just
write by hand the header and happened the cpio to it, in that case you will end
up having the same at what you are trying to do with pkg.

Bapt


pgpsclD16lX0U.pgp
Description: PGP signature


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Baptiste Daroussin
On Sun, Jul 14, 2013 at 04:52:26PM +, Teske, Devin wrote:
 
 On Jul 14, 2013, at 8:01 AM, Chris Rees wrote:
 
  On 14 Jul 2013, at 08:29, Teske, Devin wrote:
  
 
 Simple, really.
 
 Let's take RPM for example. The RPM package format has been ported to other 
 platforms.

So does pkgng ported on Linux, OS X, dragonfly, NetBSD...
 
 But, I can't take archivers/rpm4 and build on RPM on FreeBSD and install it 
 on RedHat.

Yes you can, I do it at work all the time, on FreeBSD I do create AIX rpms and
RedHat rpms.
 
 This is because the RPM format records the platform that you build your RPM 
 on (not the binaries, just the RPM) *into* said RPM.

So does pkgng.

 
 This actually adds a requirement to the RPM production that the RPMs be 
 produced on the platform that they will be installed-to.

No.
 
 Currently, no such restriction exists for the building of FreeBSD packages 
 (within our system). This would have been true if we had ported pkg_create 
 (and may continue to be true if we ported pkg and its ilk), but let's say for 
 the sake of argument that the future of pkg looks bright and it gets ported 
 to all sorts of systems (ported in a fashion similar to RPM) *and* we find 
 one day that the +MANIFEST starts containing a target-platform (resulting in 
 refusal to install a *.txz package because it was rolled on a different 
 platform.

Thank for describing the exact situation pkg is right now.

 
 In that case, we'd then prefer to by-pass the tools and use our own method of 
 creating the tar-ball to lift such a restriction.

The restriction you are speaking about does not exists.
 
 ASIDE: If I knew how to force rpmbuild into creating androgynous packages for 
 other architectures, I'd be doing that to life the restriction there too, but 
 I haven't figured out that.
 
 Basically... within our pkgbase tree, we like the branch within the tree to 
 dictate how a package is built... not what platform you're on. The goal being 
 that we can run a single package-build host that builds all of our packages 
 from a single platform.

You can do it with pkgng just easily, as well as you can do it with rpm.

Bapt


pgpUj_u2AYWpT.pgp
Description: PGP signature


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Baptiste Daroussin
On Sat, Jul 13, 2013 at 06:51:00PM -0500, Mark Felder wrote:
 On Sat, Jul 13, 2013, at 13:54, Teske, Devin wrote:
  
  
  If FTP access (or any of the other remote access methods) are going away
  for HEAD pkg access, I'll need to know so I can make the appropriate
  changes in the HEAD branch of bsdconfig.
 
 
 It's simpler than you think. The new pkg uses libfetch. You can have
 your PACAKGESITE be file:// ftp:// http:// https:// ... 
 
 I do suspect that HTTP_PROXY support is probably not available as I
 recall seeing a post where someone was asking about that getting
 implemented for fetch. I'll have to research that again, though.

Yes pkgng do support proxies, because libfetch do support them.

pkgng also support ssh://

regards,
Bapt


pgpdI3eLP7G0b.pgp
Description: PGP signature


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Baptiste Daroussin
On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote:
 On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:
 
  Hi,
  
  I have just committed (r253305) a change the make pkg_install not being 
  built
  and installed by default on HEAD.
  
  If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
  src.conf(5)
 
 I think while a good move, we need to allow a window of development to 
 re-work other HEAD components.
 
 It might also be worth developing a lint-tool to make sure we get every last 
 instance (both from C code and sh code) within our base before we motion to 
 cut a release with this change.
 
 I for one am effected and will have to change things.
 
 Can you point us at a guide (or even better, a Wiki) that can smooth the 
 process?

I have already pointed you to such resources a couple of time before you started
working on the package management part of bsdconfig. May I remember you that at
this time I have recommanded you to directly work on the pkgng version? I have
also explained you 2 or 3 times at least what is a catalog in pkgng how it works
what pkg query and pkg rquery will give you as a script writter.

Bapt


pgp2hjsox0hVc.pgp
Description: PGP signature


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Julian Elischer

On 7/15/13 6:43 AM, Craig Rodrigues wrote:

On Sat, Jul 13, 2013 at 10:54 PM, Teske, Devin devin.te...@fisglobal.comwrote:



I assume that poudiere builds packages from ports.


Yes.



  That's not how we build package repositories here (and would expect that
there are many more like us).


How do you build packages if you are not using FreeBSD ports?  Do you have
your own Makefiles which resemble FreeBSD ports,
or are you doing something completely different?



  We expect to start with an empty directory, go grab packages that we
want (one by one) from FTP. Then put those tarballs into a directory.
There's no additional step because we download the INDEX file to the
repository too. Local repository is built.


What FTP site are you grabbing packages from, the FreeBSD ftp site (or
mirrors) or one of your own?
How are you generating the INDEX file?  Do you append to the INDEX file for
every package which you sucessfully download from FTP,
or are you downloading a previously generated INDEX file?  What does your
local repository of packages consist of if you are
not using FreeBSD ports?





  But I want bsdconfig to work with local repositories without having
poudriere.



You don't have to use poudriere to build a package repository.  It is just
an optional tool that
is there for you to try out, and it works quite well when building a package
repository based off of FreeBSD ports.  If you don't want to use it, you
don't have to.





  I think so too. But right now a *lot* of unanswered questions.



Possibly, but I think that you are supporting a workflow that people on
this mailing list
don't fully understand.   If you could send out a separate e-mail
describing exactly what your workflow is, step-by-step, that might help you
get better answers
to your questions.   The FreeBSD ports team has been migrating to pkgng
for the past few years, so a lot of people understand the FreeBSD
ports/packages workflow.
because it has been discussed for a few years on multiple mailing lists, and
at various BSD conferences like BSDCan.



What's so difficult to understand? He has a local cache, populated as 
needed
from the binary packages released with the FreeBSD release, from the 
freebsd ftp site..
(He can as a supliment also add versions compiled by himself in the 
case of

security fixes etc. but that isn't required for 99% of the packages.

BSDinstall wants dependency information (currently availabele from the 
INDEX files)

to display to the user what he will get)


Some of the confusion with others on this list is that you are working on
a tool called bsdconfig and committing it into FreeBSD-CURRENT,
but the workflow you are trying to support is slightly different than what
the FreeBSD ports/packages teams have been supporting and adapting
their tools and workflow towards.

I think that if you better describe your workflow in a separate e-mail,
then folks can better recommend how to adapt bsdconfig
to the new pkg tools.  You may need to be open to using
things like libpkg inside bsdconfig, if that is the best solution for
bsdconfig inside FreeBSD.

I agree with you that having a pkgng transition document on the wiki
would be useful for systems integrators.  I know that many
people build products based on FreeBSD and leverage the pkg_XXX tools
for building their own products.  I have done this myself in the past.
Clarifying the transition path for system integrators would be very helpful.
Maybe you can help write the transition doc, since you have the perspective?

--
Craig
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org



___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Julian Elischer



On 7/15/13 10:12 AM, Teske, Devin wrote:

On Jul 14, 2013, at 3:43 PM, Craig Rodrigues wrote:

On Sat, Jul 13, 2013 at 10:54 PM, Teske, Devin 
devin.te...@fisglobal.commailto:devin.te...@fisglobal.com wrote:


I assume that poudiere builds packages from ports.

Yes.


That's not how we build package repositories here (and would expect that there 
are many more like us).

How do you build packages if you are not using FreeBSD ports?

Two spectrums...

When developer A says he needs xerces, I go to:

(for an active release, such as 8.4 or 9.1)
ftp://ftp.freebsd.org/pub/FreeBSD/releases/{arch}/X.Y-RELEASE/packages/Allftp://ftp.freebsd.org/pub/FreeBSD/releases/%7Barch%7D/X.Y-RELEASE/packages/All

(for an in-active release, such as 8.1 or 9.0)
ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/{arch}/X.Y-RELEASE/packages/Allftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/%7Barch%7D/X.Y-RELEASE/packages/All

Meanwhile, when either developer B or integrator A says I have software X that I want to 
package, we use pkgbase to build the FreeBSD package, the RedHat RPM, etc.




Before you dismiss Devin's case as corner case I will point out to 
you that:
1/your pay checks are probably just one  or two steps removed from his 
system, (almost anywhere in the world)
2/ that he is handling both FreeBSD packages AND Redhat  packages , 
and that
3/ The last time I looked his systems were handling an amount of money 
per year that had 12 '0' s in it. It's probably way bigger now.




___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Teske, Devin

On Jul 14, 2013, at 11:23 PM, Baptiste Daroussin wrote:

On Sun, Jul 14, 2013 at 10:12:19AM +, Teske, Devin wrote:

On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote:

On 14/07/2013 06:48, Teske, Devin wrote:
Question: Where can I learn more about the actual format of what's in
the new tarballs? This is going to be important not for bsdconfig,
but $work (we have our own build platform; I'm going to have to
rewrite it from mastering PLIST files to mastering YAML MANIFEST
files and I want to know all the gritty details).

We do need a pkg-manifest(5) man page, which can double as a
pkg-tarball(5) page since the manifest file is where most of the
interesting bits are.

A pkg tarball is a compressed tar archive like so:

lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz
-rw-r--r--  0 root   wheel 530 Jan  1  1970 +COMPACT_MANIFEST
-rw-r--r--  0 root   wheel6385 Jan  1  1970 +MANIFEST
-rw-r--r--  0 root   wheel   17567 Jan  1  1970 +MTREE_DIRS
-r--r--r--  0 root   wheel   19453 Jul  7 12:26
/usr/local/etc/bash_completion.d/_pkg.bash
-r-xr-xr-x  0 root   wheel 629 Jul  7 12:26
/usr/local/etc/periodic/daily/400.status-pkg
-r-xr-xr-x  0 root   wheel 823 Jul  7 12:26
/usr/local/etc/periodic/daily/411.pkg-backup
-r-xr-xr-x  0 root   wheel 678 Jul  7 12:26
/usr/local/etc/periodic/daily/490.status-pkg-changes
-r-xr-xr-x  0 root   wheel2558 Jul  7 12:26
/usr/local/etc/periodic/security/410.pkg-audit
-r-xr-xr-x  0 root   wheel 611 Jul  7 12:26
/usr/local/etc/periodic/security/460.pkg-checksum
-r--r--r--  0 root   wheel 839 Jul  7 12:26
/usr/local/etc/pkg.conf.sample
-r--r--r--  0 root   wheel   43432 Jul  7 12:26 /usr/local/include/pkg.h
-r--r--r--  0 root   wheel  727558 Jul  7 12:26 /usr/local/lib/libpkg.a
lrwxr-xr-x  0 root   wheel   0 Jul  7 12:26 /usr/local/lib/libpkg.so
- libpkg.so.1
-r--r--r--  0 root   wheel 1227064 Jul  7 12:26 /usr/local/lib/libpkg.so.1
-rw-r--r--  0 root   wheel 187 Jul  7 12:26
/usr/local/libdata/pkgconfig/pkg.pc
[... etc ...]


Interesting. I notice that (while looking ahead to see a prefix: of /usr/local 
in the +MANIFEST), the tarball itself has files that include /usr/local in 
their path.

Does pkg handle packages where the prefix (in +MANIFEST) is /usr/local _but_ 
tar tf of the unxz'd tarball is _not_ /usr/local? (e.g. pkg_install style) 
or is this the new way going forward?



There must at least be a +MANIFEST -- other meta data files are
optional.  +COMPACT_MANIFEST is a subset of the full +MANIFEST.  They're
both YAML documents.  +COMPACT_MANIFEST looks like this, for example:

---
name: pkg
version: 1.1.4
origin: ports-mgmt/pkg
comment: New generation package manager
arch: freebsd:9:x86:64
www: 
https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=LPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0As=30b27cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d
maintainer: port...@freebsd.orgmailto:port...@freebsd.org
prefix: /usr/local
licenselogic: single
licenses:
- BSD
flatsize: 6311507
desc: |
New Generation package management tool for FreeBSD

WWW: 
https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=cLKrok7Cl%2BjTLEB024lmmMXfxb1fKAOZvvxLV8RjU%2FU%3D%0As=cef93a5fe98a3fa0fbc9df3e0d11c9bbeaee44290306834ddef30f2ba925446c
categories:
- ports-mgmt
shlibs_required:
- libpkg.so.1
shlibs_provided:
- libpkg.so.1
message: |
If you are upgrading from the old package format, first run:

  # pkg2ng


Nice. Very nice.



+MTREE_DIRS is a compatibility thing with the old pkg_tools.  It's not
needed in general as +MANIFEST can provide all that meta data itself.
It isn't going to be deprecated for at least as long as the ports tree
continues to support pkg_tools though.

Beyond that, the rest of the pkg tarball just contains a tar archive of
all the files, directories, sym-links etc to be installed by the
package.  Note that pkg(8) has no problem with creating an empty
directory for a package, unlike pkg_tools.


Excellent!


Now, while you can grovel through the details of pkg tarballs and
internal data formats like this, be aware: the format of the manifest
and the details of the meta-data included in the pkg-tarballs is subject
to change without warning as we develop pkg(8) further.  We only promise
API stability through the pkg(8) commands or for the libpkg.so library
functions; reports of breakage from usage outside those APIs will
receive little sympathy.


Understood.

So just to give you a better idea of how we manage packages here.

We've realized that if you want to package for FreeBSD (in 9 and older), you 
could get by alright if you knew how to create a +CONTENTS file from scratch. 
For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an awful lot 
like a SPECFILE (they are both in-fact YAML).

The only thing you need if you 

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Teske, Devin

On Jul 14, 2013, at 11:27 PM, Baptiste Daroussin wrote:

 On Sun, Jul 14, 2013 at 07:29:50AM +, Teske, Devin wrote:
 
 Or you
 can maintain the old cruft for your business -- just don't expect
 anyone else to use it, or even want to.
 
 
 
 I have no intention of making old-world packages... but I also have no 
 intention of using pkg create.
 
 So you are taking as an exemple rpm but with RPM you do want to use what they 
 do
 provide, aka rpmbuild but with pkgng you want to manually handwrite the 
 packages
 because we haven't hidden the format behind opaque names and format.
 

Close... I use rpmbuild because I haven't figured out how to subvert it to 
create cpio-balls for rpm(8). I know how to extract the contents of binary 
rpm's using rpm2cpio and I know how to extract the meta data and supplant it 
into a template SPECFILE using rpm -q -p file.rpm --qf queryformat ... but I 
haven't spent enough time with rpm's to know how to master them without 
rpmbuild. The best I've done is learn how to use .rpmmacros to make rpmbuild 
behave in a sane manner that is conducive to (a) building the archive out of 
the current directory (including all temporary files, etc.) and (b) working on 
source-controlled checkouts.

However, even if I did figure out the cpio-ball/metadata format, it is quite 
convenient to have rpmbuild do the work -- which unfortunately introduces the 
constraint that you have to generate the archive on the system for which it 
will be installed (something you eluded that is inconclusive -- I'll cover that 
in a reply to that-reply directly).

The convenience however is not because the tool does everything I need it to 
do, but the convenience comes from the translation of YAML into metadata. It 
doesn't appear that this will be necessary with pkgng packages, just as it 
wasn't necessary for pkg_install packages.

To be more clear,... because tools like pkg_create and pkg create take a 
file-in, add data to it, and then (the most important part) put a modified form 
into the archive (which quite-closely resembles the original), it is then 
possible to imagine that:

(1) there's value in learning both the input format and the output format so 
that you can validate packages, fix broken packages, etc.

ASIDE: For example... a binary release package ends up with incorrect 
dependencies encoded within its +CONTENTS file -- I can slurp the package into 
our pkgbase, adjust the PLIST (generated from +CONTENTS), and produce a fixed 
package in mere moments (and I didn't have to wake anybody in the middle of the 
night, ask that FTP mirrors be updated, balk at the core team -- none of 
that... the problem was solved and in the next release we hope for the best 
that the dependency chain isn't busted for the same binary package within the 
set of X.Y-RELEASE packages).

(2) there's value in writing a shell-script method to translate the input 
format into the output format from a system of makefiles calling said shell 
scripts out of a revision-controlled checkout

ASIDE: This one should be obvious. If you can assemble a package using nothing 
but shell scripts (and mostly awk, ls, find, cp, mkdir, ln, cvs, make, 
md5/sha256/sha512/openssl, and etc.), then you don't have to port the port 
system (our port system being called pkgbase -- and working to keep you 
stable rather than keep you up-to-date).



 You know in that case you should probably create your own RPM without RPM just
 write by hand the header and happened the cpio to it, in that case you will 
 end
 up having the same at what you are trying to do with pkg.
 

It's a good idea... except one thing... the YAML SPECFILE (minus the elements 
that are specific to the `-bb' process of rpmbuild) get completely changed in 
the process. Doing this would require a YAML parser + metadata encoder 
process-pair. Whereas for FreeBSD it's YAML-in, YAML-out. So I don't need a 
YAML-parser, just a line-parser and light YAML-encoder -- which could be 
fixed-strings, doesn't have to be complex... e.g., awk).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Teske, Devin

On Jul 14, 2013, at 11:33 PM, Baptiste Daroussin wrote:

On Sun, Jul 14, 2013 at 04:52:26PM +, Teske, Devin wrote:

On Jul 14, 2013, at 8:01 AM, Chris Rees wrote:

On 14 Jul 2013, at 08:29, Teske, Devin wrote:


Simple, really.

Let's take RPM for example. The RPM package format has been ported to other 
platforms.

So does pkgng ported on Linux, OS X, dragonfly, NetBSD...

Sweet! (bright future!)



But, I can't take archivers/rpm4 and build on RPM on FreeBSD and install it on 
RedHat.

Yes you can, I do it at work all the time, on FreeBSD I do create AIX rpms and
RedHat rpms.

What version of RedHat?

Worked on RedHat EL4 using rpm3, but in-practice -- attempts to recreate that 
workflow on RedHat EL6 using rpm4 have failed.

Please see a copy/paste of the output of when we build an RPM on FreeBSD-8.1 
and try to install it on RedHat Enterprise Linux 6:

http://pastebin.com/zpzjxP2T

Spoiler: package {X} is intended for a freebsd operating system (not 
installed)





This is because the RPM format records the platform that you build your RPM 
on (not the binaries, just the RPM) *into* said RPM.

So does pkgng.


Good to know.



This actually adds a requirement to the RPM production that the RPMs be 
produced on the platform that they will be installed-to.

No.

Yes. See pastebin link above.




Currently, no such restriction exists for the building of FreeBSD packages 
(within our system). This would have been true if we had ported pkg_create (and 
may continue to be true if we ported pkg and its ilk), but let's say for the 
sake of argument that the future of pkg looks bright and it gets ported to 
all sorts of systems (ported in a fashion similar to RPM) *and* we find one day 
that the +MANIFEST starts containing a target-platform (resulting in refusal to 
install a *.txz package because it was rolled on a different platform.

Thank for describing the exact situation pkg is right now.

Glad to help ;D

(not so happy about the target platform being recorded -- is there an override? 
setting UNAME_{a,r,etc.}?)




In that case, we'd then prefer to by-pass the tools and use our own method of 
creating the tar-ball to lift such a restriction.

The restriction you are speaking about does not exists.

See pastebin link above.



ASIDE: If I knew how to force rpmbuild into creating androgynous packages for 
other architectures, I'd be doing that to life the restriction there too, but I 
haven't figured out that.

Basically... within our pkgbase tree, we like the branch within the tree to 
dictate how a package is built... not what platform you're on. The goal being 
that we can run a single package-build host that builds all of our packages 
from a single platform.

You can do it with pkgng just easily, as well as you can do it with rpm.


W/respect to RPM, see pastebin link above.
--
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Teske, Devin

On Jul 14, 2013, at 11:38 PM, Baptiste Daroussin wrote:

 On Sat, Jul 13, 2013 at 06:51:00PM -0500, Mark Felder wrote:
 On Sat, Jul 13, 2013, at 13:54, Teske, Devin wrote:
 
 
 If FTP access (or any of the other remote access methods) are going away
 for HEAD pkg access, I'll need to know so I can make the appropriate
 changes in the HEAD branch of bsdconfig.
 
 
 It's simpler than you think. The new pkg uses libfetch. You can have
 your PACAKGESITE be file:// ftp:// http:// https:// ... 
 
 I do suspect that HTTP_PROXY support is probably not available as I
 recall seeing a post where someone was asking about that getting
 implemented for fetch. I'll have to research that again, though.
 
 Yes pkgng do support proxies, because libfetch do support them.
 

I think Mark was saying that libfetch doesn't yet support http proxy.

However, I've often done:

env http_proxy=user:pass@host:port fetch this-thing or that-thing

So I'm not sure if Mark was implying that this doesn't yet work with 
PACKAGESITE because you need a way of passing the additional env? (talk of a 
PKG_ENV iirc)?


 pkgng also support ssh://
 

That's cool ;D
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Mark Felder
On Mon, 15 Jul 2013 11:48:18 -0500, Teske, Devin
devin.te...@fisglobal.com wrote:
 I think Mark was saying that libfetch doesn't yet support http proxy.

Now that I've googled, I was referring to this PR
http://www.freebsd.org/cgi/query-pr.cgi?pr=180253

I thought it was complete missing functionality, not just that bug.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-15 Thread Teske, Devin

On Jul 14, 2013, at 11:42 PM, Baptiste Daroussin wrote:

 On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote:
 On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:
 
 Hi,
 
 I have just committed (r253305) a change the make pkg_install not being 
 built
 and installed by default on HEAD.
 
 If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
 src.conf(5)
 
 I think while a good move, we need to allow a window of development to 
 re-work other HEAD components.
 
 It might also be worth developing a lint-tool to make sure we get every last 
 instance (both from C code and sh code) within our base before we motion to 
 cut a release with this change.
 
 I for one am effected and will have to change things.
 
 Can you point us at a guide (or even better, a Wiki) that can smooth the 
 process?
 
 I have already pointed you to such resources a couple of time before you 
 started
 working on the package management part of bsdconfig.

Respectfully... look at the reply-text. I didn't mention bsdconfig. I was 
asking in-general for others. I think we're all in agreement that a transition 
Wiki does not exist. I'm not upset that the conversation was turned into a 
bsdconfig-specific one (on the contrary, elated)... but for the very reason 
that you had already given me pointers in the past, I did not come right out 
and say hey, could you re-tell me everything you told me before because I've 
forgotten most of it? but instead I pointed out that we don't have a 
transition Wiki.

So I do supremely apologize if I ask a question that you already answered in 
the past.

My excuse is that you pointed me to such resources almost 10 months ago. And 
for that entire 10 months, I'd been working on the package management part of 
bsdconfig. I *just* got my head above water and then it was announced that 
pkg_install is going away, and what perfect timing -- having just finished the 
pkg_install-based framework for which I plan to base the pkgng integration.


 May I remember you that at
 this time I have recommanded you to directly work on the pkgng version?

This comment doesn't seem to be related to any of the reply-text above it. I'll 
assume it's tainted by the other messages in this thread. For bsdconfig, let me 
just re-iterate...

+ bsdconfig is the tool to replace sysinstall configPackages.
+ It will not do anything weird or strange or out of place **
+ It will try to be an example as well as a re-usable and extendable tool

(and on the back of those last two comments, again, bsdconfig pkgng integration 
is not associated with any funky handling of package MANIFEST files or any 
black-magic voodoo that I do at $work with pkgbase).


** Unless a UI enhancement can be made by subverting a decidedly non-UI tool to 
provide more feedback than simply blocking on an opaque process.


 I have
 also explained you 2 or 3 times at least what is a catalog in pkgng how it 
 works
 what pkg query and pkg rquery will give you as a script writter.
 

Again, apologies if I ask a question that *seems* like it's been asked before.

And it would be a lie for me to say I've forgotten *everything* from our prior 
conversations. However, I may repeat some questions as it was almost 10 months 
ago that I was last looking at pkgng (getting bsdconfig packages done for 
9.2-R was a *lot* of work and it depleted my brain of pretty much everything 
else for that entire time -- except for $work of course, but every night for 10 
months, 7 days a week, I ate, drank, slept, and smoked bsdconfig packages).

If I could recount something from our conversations from 10 months ago... I 
would say that you and I agreed that I needed to get the framework done and 
that I could best do that unencumbered if it were pkg_install based. Well 
that's been done (it was a hell of a lot of work, but I got it done).

I'm saying that one sole request (to get the framework ready for pkgng) is now 
done and we need to reconvene to take the next step.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Teske, Devin

On Jul 13, 2013, at 11:13 PM, Garrett Wollman wrote:

 In article 20130714054840$7...@grapevine.csail.mit.edu,
 dte...@freebsd.org writes:
 
 How about rquery? What protocol does that use? and what does it talk to?
 
 It accesses the sqlite database in /var/db/pkg that was previously
 retrieved from the remote repository.
 

This previously retrieved has me worried.

One of the things that you could do with sysinstall was:

1. Browse packages (nothing else)
2. Exit sysinstall

And your system is completely unmodified and the mirror your chose is forgotten.

Now from what you explained of pkg, I'm worried that for bsdconfig:

1. Browse packages (nothing else)
2. Exit bsdconfig

and now because the pkg rquery has staged a db, future pkg commands are now 
influenced.

In which case, I think I would best off downloading the sqlite db to a 
temporary location and pointing pkg at it (if that's possible). Because I'm not 
sure that I want adhoc browsing in bsdconfig to change the default mirror (but 
maybe that's desired? I'm not sure).



 Question: Where can I learn more about the actual format of what's in
 the new tarballs? This is going to be important not for bsdconfig, but
 $work (we have our own build platform; I'm going to have to rewrite it
 from mastering PLIST files to mastering YAML MANIFEST files and I want
 to know all the gritty details).
 
 You really shouldn't know about the actual format of the tarballs;
 your time would be better spent learning the new pkg create, pkg
 register, and pkg repo commands.  Depending on your needs, you
 might want to write to the libpkg API instead.
 

That won't work for us.

You're not going to like the answer, but it does mean that things like pkg 
create, pkg register and pkg repo won't work for us.

Because FreeBSD packages are just tar-balls, I created a build-system that 
allows us to master and re-master them on *any* platform with nothing more than 
make and a few standard UNIX utilities such as awk and etc. This makes it 
extremely convenient for developers and integrators. You can feel free to check 
out the build tree on Mac OS X, make some changes to a file, check them in, 
build the package, throw it in the repos, and all from the comfort of your own 
OS.

Yes, it flies in the face of dog-fooding, but the value-add that it brings to 
the table is quite tangible w/respect to our workflow.

So the format is still something I want to know quite well.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Teske, Devin

On Jul 14, 2013, at 12:06 AM, Garrett Wollman wrote:

 In article 20130714064601$3...@grapevine.csail.mit.edu,
 dte...@freebsd.org writes:
 
 [I wrote:]
 It accesses the sqlite database in /var/db/pkg that was previously
 retrieved from the remote repository.
 
 Now from what you explained of pkg, I'm worried that for bsdconfig:
 
 1. Browse packages (nothing else)
 2. Exit bsdconfig
 
 and now because the pkg rquery has staged a db, future pkg commands
 are now influenced.
 
 Only if you update /usr/local/etc/pkg.conf to set a permanent
 PACKAGESITE.  Otherwise, it will notice that the remote repo catalog
 you have isn't for your currently selected remote repo, and use that
 instead.
 
 You really shouldn't know about the actual format of the tarballs;
 your time would be better spent learning the new pkg create, pkg
 register, and pkg repo commands.  Depending on your needs, you
 might want to write to the libpkg API instead.
 
 That won't work for us.
 
 You're not going to like the answer, but it does mean that things like
 pkg create, pkg register and pkg repo won't work for us.
 
 Congratulations for building your entire workflow around a horrible
 kluge straight out of 1993.

Prejudice much?


  FreeBSD, however, is moving on.

Moving from tarballs to tarballs, yep... moving along nicely.


  (And
 it's long past time!)

I love your enthusiastic embrace.



  Your developers will just have to deal.

You're making the wrong argument.

Bringing in a new package system doesn't help developers or integrators. It 
doesn't matter if you switch pkg_create with pkg_create. Developers like 
being able to go into a build tree and say make and end-up with a package.

How that build tree does it should be sufficiently advanced enough to be 
indistinguishable from magic.

Your answer implies that this infrastructure is unnecessary, when I can indeed 
tell you that even if it is not divulged to me what goes into a package, I'll 
still end up ripping it open and creating my own build platform that doesn't 
depend on platform-specific tools.

And when the day comes that FreeBSD uses something other than tarballs, then 
I'll then force the developers to build the packages on the platform using said 
tools; but until then, why pigeon-hole the process of building a package into 
one OS when developers could care less whether their make uses pkg_create, 
pkg, rpm, dpkg, or what else?

To give you an idea as to just how helpful this is...

Imagine the following hierarchy:

src/pkgbase/depend/mystuff/script1
src/pkgbase/depend/mystuff/textfile1
src/pkgbase/depend/mystuff/sourcefile.c
src/pkgbase/depend/mystuff/Makefile

You are a developer. You want to ship a package that contains script1, 
textfile1, and binary1 (which is compiled by saying make to turn 
sourcefile.c into binary1)

You want to ship 8 types of packages:

FreeBSD-4.11
FreeBSD-8.1 (i386)
FreeBSD-8.1 (amd64)
RedHat EL 4
RedHat EL 6 (i386)
RedHat EL 6 (x86_64)
Debian Wheezy
Debian Wheezy 64-bit

This is where my framework comes in-handy...

cd ~/src/pkgbase/freebsd/RELENG_4/category/mystuff
make
# it pulled the necessary bits from src/pkgbase/depend/mystuff and built the 
.tgz

cd ~/src/pkgbase/freebsd/RELENG_8/category/mystuff
make
# it pulled the necessary bits from the depend dir and built .tbz

cd ~/src/pkgbase/redhat/rhel4/category/sub-category/mystuff
make
# pulled in depend and made .rpm

cd ~/src/pkgbase/redhat/rhel6/category/sub-category/mystuff
make
# pulled in depend and made .rpm

etc.

Of course, *any* time the depend tree has binaries in it... you have to first 
do a make in there on the platform you want to ship the binary for, and then do 
make depend in the platform-specific tree to pull in the binaries. Once 
you've done that, you don't have to muck with the depend tree again unless 
there are changes there.

So, I assume that your prejudice remarks are because you haven't either seen 
(a) such a platform or (b) such a need for said platform.

Yeah, I could rewrite the freebsd-specific logic to use pkg create, but let 
me tell you...

When you have to touch a file that needs to get shipped out to multiple 
platforms...

It's damned nice to be able to build the FreeBSD packages under RedHat 
*BECAUSE* the redhat RPMs can't be built under anything else (building an RPM 
on FreeBSD and attempting to install it on RedHat results in an error message 
similar to this is an rpm for FreeBSD; go away).

Whereas FreeBSD will never balk about a package built on another platform.

It's a huge time-saving measure... not having to jump over to each/every unique 
platform to package things up *IF/WHEN* you know that there are no binaries in 
the package *or* you've already checked the pre-compiled binaries into the 
arch-specific hierarchy.




  Or you
 can maintain the old cruft for your business -- just don't expect
 anyone else to use it, or even want to.
 


I have no intention of making old-world packages... but I also have no 
intention of using 

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Matthew Seaman
On 14/07/2013 04:18, Teske, Devin wrote:

 ASIDE: For efficiency, I will actually need three things: (1) a list
 of all packages (2) their descriptions and (3) their run-time dependencies.

That would be the repository catalogue, which you can download by 'pkg
update' (or it will happen automatically before pkg operations.)  Or,
more precisely, the repository /catalogues/ for each of the package
repositories enabled in pkg.conf

 NOTE: I'll need run-time dependencies so that as they're checking
 boxes in the UI, I can update accordingly (has nothing to do with how
 the dependencies are handled when the packages get installed; I'll let
 pkg handle that when it comes time, but for the UI and for the review
 screen, I want the user to be informed and I can do that more
 efficiently if I have a master-list and don't have to make continuous
 queries).

In 1.1 the repository catalogue is just the collection of all the YAML
format metadata from all the packages in the repository, plus optional
cryptographic signatures.  While you could try and parse that directly,
there's little point in re-inventing that wheel -- pkg will load the
catalogue data into a sqlite database, and it provides 'pkg search' or
'pkg rquery' specifically for searching catalogue data. ('pkg rquery' is
aimed at scripting use, 'pkg search' is more for interactive use.)

 That being said, I was planning on doing a pkg rquery to get that
 master list of [minimally] 3-pieces of information.

Something like:

pkg rquery -a '%n-%v %o\n%e\n%dn-%dv %do\n---\n'

 What protocol does that rquery run on? What would one have to do
 to set up their own server that can accept an rquery? (our customers
 don't have Internet access)

It's built around SQL queries against the sqlite database built from the
repository catalogues.  Once you've got the repo catalgoue, it doesn't
need network access to just query the data.

 Last but not least...
 
 Can you do an rquery on a local repository? (say, one that has
 been mounted via NFS or some other media, local or otherwise but looking like
 a local filesystem once-mounted). What would be required to get a local
 repository like that going?

You can certainly build a local repository -- essentially you just
assemble a set of packages in some sort of directory heirarchy, and then
just run 'pkg repo' to create the catalogue.

You can then access the repo by any of the protocols supported by
libfetch -- which includes file://, suitable for a repo on a local or
NFS mounted directory.  Beyond what libfetch provides, you can also use
ssh://.

You'ld usually use poudriere or similar to build your own set of
packages to populate your private repository, but there's nothing to
stop you eg. mirroring some selection of packages from a public package
repository and building your private repo from them.  Or configuring
your systems to use a local package repository /and/ a public
repository.  See pkg-repository(5).

 I do suspect that HTTP_PROXY support is probably not available as I
 recall seeing a post where someone was asking about that getting
 implemented for fetch. I'll have to research that again, though.

 Thanks. Keep me up to date on that.

pkg will use the proxing capabilities of libfetch -- so it will abide by
any HTTP_PROXY or FTP_PROXY or any of a number of other environment
variables as described in fetch(3).  You can use the PKG_ENV setting in
pkg.conf(5) to set the environment use by pkg commands.

 Neither sysinstall nor bsdinstall really give HTTP_PROXY access much
 thought (they support it, but only minimally). They just construct a
 raw HTTP header with the FTP URL and send that directly to the proxy.
 One cute thing it does (when initializing the connection) is test for
 Squid and if-so, appends ;type=i to the URL (to force binary
 download versus ascii).

Everything that pkg downloads is currently in the form of an xz(1)
compressed tar archive[*] so definitely needs to downloaded in binary mode.

[*] Even in some cases where the tarball contains only one file, which
is a bit of a nonsense, but not hugely so.  That may well be changed in
future versions.

 No support for proxy-server authentication (however user/pass
 authentication for the FTP server is passed through to the
 proxy-server).

pkg uses libfetch's proxy auth support.  You can also authenticate
access to a repository via FTP or HTTP or HTTPS by encoding a username
and password in the repository URL in the usual way.  For access over
SSH, use any of the authentication mechanisms provided by the ssh server
-- use of ssh-agent(1) and key-based auth is recommended, to avoid
having to type in passphrases repeatedly.

I think if you pkgng-ify a test system and have a play with the
capabilities pkg(8) provides (and maybe poudriere(8) too), you will be
pleased, maybe even pleasantly surprised, by what it can do.

Cheers,

Matthew

-- 
Dr Matthew J Seaman MA, D.Phil.
PGP: http://www.infracaninophile.co.uk/pgpkey




signature.asc

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Teske, Devin

On Jul 13, 2013, at 9:24 PM, Peter Wemm wrote:

 On Sat, Jul 13, 2013 at 11:54 AM, Teske, Devin
 devin.te...@fisglobal.com wrote:
 
 So yes... I'm asking... in a HEAD world, what is the officially supported 
 method of acquisition?
 
 * pkgng will be configured to pull via http from project
 infrastructure by default.

So is this the right _first step_ patch for HEAD bsdconfig?

NOTE: Based on feedback, this will be the list of what official pkg 
infrastructure we want to support from the base. The selection would be 
communicated to pkg via $PACKAGESITE.

Index: head/usr.sbin/bsdconfig/share/media/http.subr
===
--- head/usr.sbin/bsdconfig/share/media/http.subr   (revision 25)
+++ head/usr.sbin/bsdconfig/share/media/http.subr   (working copy)
@@ -77,7 +77,9 @@ f_dialog_menu_media_http()
f_dialog_title_restore
local prompt=$msg_please_select_the_site_closest_to_you_or_other
local menu_list=
-   'URL' '$msg_specify_some_other_http_site'
+   '$msg_main_site'  'http.freebsd.org'
+   'URL' '$msg_specify_some_other_http_site'
+   'IPv6 $msg_main_site' 'http.freebsd.org'
 # END-QUOTE
local hline=$msg_select_a_site_thats_close
 
Index: head/usr.sbin/bsdconfig/share/media/ftp.subr
===
--- head/usr.sbin/bsdconfig/share/media/ftp.subr(revision 25)
+++ head/usr.sbin/bsdconfig/share/media/ftp.subr(working copy)
@@ -73,157 +73,7 @@ f_dialog_menu_media_ftp()
f_dialog_title_restore
local prompt=$msg_please_select_the_site_closest_to_you_or_other
local menu_list=
-   '$msg_main_site'  'ftp.freebsd.org'
-   'URL' '$msg_specify_some_other_ftp_site'
-   '$msg_snapshots_server_japan'
- 'snapshots.jp.freebsd.org'
-   '$msg_snapshots_server_sweden'
- 'snapshots.se.freebsd.org'
-   'IPv6 $msg_main_site' 'ftp.freebsd.org'
-   ' IPv6 $msg_ireland'  'ftp3.ie.freebsd.org'
-   ' IPv6 $msg_israel'   'ftp.il.freebsd.org'
-   ' IPv6 $msg_japan''ftp2.jp.freebsd.org'
-   ' IPv6 $msg_usa'  'ftp4.us.freebsd.org'
-   ' IPv6 $msg_turkey'   'ftp2.tr.freebsd.org'
-   '$msg_primary''ftp1.freebsd.org'
-   ' $msg_primary #2''ftp2.freebsd.org'
-   ' $msg_primary #3''ftp3.freebsd.org'
-   ' $msg_primary #4''ftp4.freebsd.org'
-   ' $msg_primary #5''ftp5.freebsd.org'
-   ' $msg_primary #6''ftp6.freebsd.org'
-   ' $msg_primary #7''ftp7.freebsd.org'
-   ' $msg_primary #8''ftp8.freebsd.org'
-   ' $msg_primary #9''ftp9.freebsd.org'
-   ' $msg_primary #10'   'ftp10.freebsd.org'
-   ' $msg_primary #11'   'ftp11.freebsd.org'
-   ' $msg_primary #12'   'ftp12.freebsd.org'
-   ' $msg_primary #13'   'ftp13.freebsd.org'
-   ' $msg_primary #14'   'ftp14.freebsd.org'
-   '$msg_argentina'  'ftp.ar.freebsd.org'
-   '$msg_australia'  'ftp.au.freebsd.org'
-   ' $msg_australia #2'  'ftp2.au.freebsd.org'
-   ' $msg_australia #3'  'ftp3.au.freebsd.org'
-   '$msg_austria''ftp.at.freebsd.org'
-   ' $msg_austria #2''ftp2.at.freebsd.org'
-   '$msg_brazil' 'ftp.br.freebsd.org'
-   ' $msg_brazil #2' 'ftp2.br.freebsd.org'
-   ' $msg_brazil #3' 'ftp3.br.freebsd.org'
-   ' $msg_brazil #4' 'ftp4.br.freebsd.org'
-   ' $msg_brazil #5' 'ftp5.br.freebsd.org'
-   ' $msg_brazil #6' 'ftp6.br.freebsd.org'
-   ' $msg_brazil #7' 'ftp7.br.freebsd.org'
-   '$msg_canada' 'ftp.ca.freebsd.org'
-   '$msg_china'  'ftp.cn.freebsd.org'
-   ' $msg_china #2'  'ftp2.cn.freebsd.org'
-   '$msg_croatia''ftp.hr.freebsd.org'
-   '$msg_czech_republic' 'ftp.cz.freebsd.org'
-   '$msg_denmark''ftp.dk.freebsd.org'
-   ' $msg_denmark #2''ftp2.dk.freebsd.org'
-   '$msg_estonia''ftp.ee.freebsd.org'
-   '$msg_finland''ftp.fi.freebsd.org'
-   '$msg_france' 'ftp.fr.freebsd.org'
-   ' $msg_france #2' 'ftp2.fr.freebsd.org'
-   ' $msg_france #3' 'ftp3.fr.freebsd.org'
-   ' 

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Matthew Seaman
On 14/07/2013 06:48, Teske, Devin wrote:
 Question: Where can I learn more about the actual format of what's in
 the new tarballs? This is going to be important not for bsdconfig,
 but $work (we have our own build platform; I'm going to have to
 rewrite it from mastering PLIST files to mastering YAML MANIFEST
 files and I want to know all the gritty details).

We do need a pkg-manifest(5) man page, which can double as a
pkg-tarball(5) page since the manifest file is where most of the
interesting bits are.

A pkg tarball is a compressed tar archive like so:

lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz
-rw-r--r--  0 root   wheel 530 Jan  1  1970 +COMPACT_MANIFEST
-rw-r--r--  0 root   wheel6385 Jan  1  1970 +MANIFEST
-rw-r--r--  0 root   wheel   17567 Jan  1  1970 +MTREE_DIRS
-r--r--r--  0 root   wheel   19453 Jul  7 12:26
/usr/local/etc/bash_completion.d/_pkg.bash
-r-xr-xr-x  0 root   wheel 629 Jul  7 12:26
/usr/local/etc/periodic/daily/400.status-pkg
-r-xr-xr-x  0 root   wheel 823 Jul  7 12:26
/usr/local/etc/periodic/daily/411.pkg-backup
-r-xr-xr-x  0 root   wheel 678 Jul  7 12:26
/usr/local/etc/periodic/daily/490.status-pkg-changes
-r-xr-xr-x  0 root   wheel2558 Jul  7 12:26
/usr/local/etc/periodic/security/410.pkg-audit
-r-xr-xr-x  0 root   wheel 611 Jul  7 12:26
/usr/local/etc/periodic/security/460.pkg-checksum
-r--r--r--  0 root   wheel 839 Jul  7 12:26
/usr/local/etc/pkg.conf.sample
-r--r--r--  0 root   wheel   43432 Jul  7 12:26 /usr/local/include/pkg.h
-r--r--r--  0 root   wheel  727558 Jul  7 12:26 /usr/local/lib/libpkg.a
lrwxr-xr-x  0 root   wheel   0 Jul  7 12:26 /usr/local/lib/libpkg.so
- libpkg.so.1
-r--r--r--  0 root   wheel 1227064 Jul  7 12:26 /usr/local/lib/libpkg.so.1
-rw-r--r--  0 root   wheel 187 Jul  7 12:26
/usr/local/libdata/pkgconfig/pkg.pc
[... etc ...]

There must at least be a +MANIFEST -- other meta data files are
optional.  +COMPACT_MANIFEST is a subset of the full +MANIFEST.  They're
both YAML documents.  +COMPACT_MANIFEST looks like this, for example:

---
name: pkg
version: 1.1.4
origin: ports-mgmt/pkg
comment: New generation package manager
arch: freebsd:9:x86:64
www: http://wiki.freebsd.org/pkgng
maintainer: port...@freebsd.org
prefix: /usr/local
licenselogic: single
licenses:
- BSD
flatsize: 6311507
desc: |
  New Generation package management tool for FreeBSD

  WWW: http://wiki.freebsd.org/pkgng
categories:
- ports-mgmt
shlibs_required:
- libpkg.so.1
shlibs_provided:
- libpkg.so.1
message: |
  If you are upgrading from the old package format, first run:

# pkg2ng

+MTREE_DIRS is a compatibility thing with the old pkg_tools.  It's not
needed in general as +MANIFEST can provide all that meta data itself.
It isn't going to be deprecated for at least as long as the ports tree
continues to support pkg_tools though.

Beyond that, the rest of the pkg tarball just contains a tar archive of
all the files, directories, sym-links etc to be installed by the
package.  Note that pkg(8) has no problem with creating an empty
directory for a package, unlike pkg_tools.

Now, while you can grovel through the details of pkg tarballs and
internal data formats like this, be aware: the format of the manifest
and the details of the meta-data included in the pkg-tarballs is subject
to change without warning as we develop pkg(8) further.  We only promise
API stability through the pkg(8) commands or for the libpkg.so library
functions; reports of breakage from usage outside those APIs will
receive little sympathy.

Cheers,

Matthew

-- 
Dr Matthew J Seaman MA, D.Phil.
PGP: http://www.infracaninophile.co.uk/pgpkey




signature.asc
Description: OpenPGP digital signature


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Teske, Devin

On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote:

 On 14/07/2013 06:48, Teske, Devin wrote:
 Question: Where can I learn more about the actual format of what's in
 the new tarballs? This is going to be important not for bsdconfig,
 but $work (we have our own build platform; I'm going to have to
 rewrite it from mastering PLIST files to mastering YAML MANIFEST
 files and I want to know all the gritty details).
 
 We do need a pkg-manifest(5) man page, which can double as a
 pkg-tarball(5) page since the manifest file is where most of the
 interesting bits are.
 
 A pkg tarball is a compressed tar archive like so:
 
 lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz
 -rw-r--r--  0 root   wheel 530 Jan  1  1970 +COMPACT_MANIFEST
 -rw-r--r--  0 root   wheel6385 Jan  1  1970 +MANIFEST
 -rw-r--r--  0 root   wheel   17567 Jan  1  1970 +MTREE_DIRS
 -r--r--r--  0 root   wheel   19453 Jul  7 12:26
 /usr/local/etc/bash_completion.d/_pkg.bash
 -r-xr-xr-x  0 root   wheel 629 Jul  7 12:26
 /usr/local/etc/periodic/daily/400.status-pkg
 -r-xr-xr-x  0 root   wheel 823 Jul  7 12:26
 /usr/local/etc/periodic/daily/411.pkg-backup
 -r-xr-xr-x  0 root   wheel 678 Jul  7 12:26
 /usr/local/etc/periodic/daily/490.status-pkg-changes
 -r-xr-xr-x  0 root   wheel2558 Jul  7 12:26
 /usr/local/etc/periodic/security/410.pkg-audit
 -r-xr-xr-x  0 root   wheel 611 Jul  7 12:26
 /usr/local/etc/periodic/security/460.pkg-checksum
 -r--r--r--  0 root   wheel 839 Jul  7 12:26
 /usr/local/etc/pkg.conf.sample
 -r--r--r--  0 root   wheel   43432 Jul  7 12:26 /usr/local/include/pkg.h
 -r--r--r--  0 root   wheel  727558 Jul  7 12:26 /usr/local/lib/libpkg.a
 lrwxr-xr-x  0 root   wheel   0 Jul  7 12:26 /usr/local/lib/libpkg.so
 - libpkg.so.1
 -r--r--r--  0 root   wheel 1227064 Jul  7 12:26 /usr/local/lib/libpkg.so.1
 -rw-r--r--  0 root   wheel 187 Jul  7 12:26
 /usr/local/libdata/pkgconfig/pkg.pc
 [... etc ...]
 

Interesting. I notice that (while looking ahead to see a prefix: of /usr/local 
in the +MANIFEST), the tarball itself has files that include /usr/local in 
their path.

Does pkg handle packages where the prefix (in +MANIFEST) is /usr/local _but_ 
tar tf of the unxz'd tarball is _not_ /usr/local? (e.g. pkg_install style) 
or is this the new way going forward?



 There must at least be a +MANIFEST -- other meta data files are
 optional.  +COMPACT_MANIFEST is a subset of the full +MANIFEST.  They're
 both YAML documents.  +COMPACT_MANIFEST looks like this, for example:
 
 ---
 name: pkg
 version: 1.1.4
 origin: ports-mgmt/pkg
 comment: New generation package manager
 arch: freebsd:9:x86:64
 www: 
 https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgngk=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=LPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0As=30b27cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d
 maintainer: port...@freebsd.org
 prefix: /usr/local
 licenselogic: single
 licenses:
 - BSD
 flatsize: 6311507
 desc: |
  New Generation package management tool for FreeBSD
 
  WWW: http://wiki.freebsd.org/pkgng
 categories:
 - ports-mgmt
 shlibs_required:
 - libpkg.so.1
 shlibs_provided:
 - libpkg.so.1
 message: |
  If you are upgrading from the old package format, first run:
 
# pkg2ng
 

Nice. Very nice.



 +MTREE_DIRS is a compatibility thing with the old pkg_tools.  It's not
 needed in general as +MANIFEST can provide all that meta data itself.
 It isn't going to be deprecated for at least as long as the ports tree
 continues to support pkg_tools though.
 
 Beyond that, the rest of the pkg tarball just contains a tar archive of
 all the files, directories, sym-links etc to be installed by the
 package.  Note that pkg(8) has no problem with creating an empty
 directory for a package, unlike pkg_tools.
 

Excellent!


 Now, while you can grovel through the details of pkg tarballs and
 internal data formats like this, be aware: the format of the manifest
 and the details of the meta-data included in the pkg-tarballs is subject
 to change without warning as we develop pkg(8) further.  We only promise
 API stability through the pkg(8) commands or for the libpkg.so library
 functions; reports of breakage from usage outside those APIs will
 receive little sympathy.
 

Understood.

So just to give you a better idea of how we manage packages here.

We've realized that if you want to package for FreeBSD (in 9 and older), you 
could get by alright if you knew how to create a +CONTENTS file from scratch. 
For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an awful lot 
like a SPECFILE (they are both in-fact YAML).

So rather than teach the people here how to use tools, I taught them what I 
think is more important... how to manage +CONTENTS, SPECFILE, and now 
up-and-coming, +MANIFEST.

However, I'd be lying if I said I taught them *all* the gory details. In 
reality, for +CONTENTS they edit a PLIST which is essentially +CONTENTS 

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Garrett Wollman
In article 20130714054840$7...@grapevine.csail.mit.edu,
dte...@freebsd.org writes:

How about rquery? What protocol does that use? and what does it talk to?

It accesses the sqlite database in /var/db/pkg that was previously
retrieved from the remote repository.

Question: Where can I learn more about the actual format of what's in
the new tarballs? This is going to be important not for bsdconfig, but
$work (we have our own build platform; I'm going to have to rewrite it
from mastering PLIST files to mastering YAML MANIFEST files and I want
to know all the gritty details).

You really shouldn't know about the actual format of the tarballs;
your time would be better spent learning the new pkg create, pkg
register, and pkg repo commands.  Depending on your needs, you
might want to write to the libpkg API instead.

-GAWollman

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Matthew Seaman
On 14/07/2013 11:12, Teske, Devin wrote:

 Interesting. I notice that (while looking ahead to see a prefix: of
 /usr/local in the +MANIFEST), the tarball itself has files that include
 /usr/local in their path.

Yes -- we consider the $PREFIX to be 'baked into' the package at compile
time.  You can relocate a package to some extent at installation time --
so for instance you can install into a jail or chroot from outside it --
but the software would expect all the files to be arranged as shown in
the package at runtime.

 Does pkg handle packages where the prefix (in +MANIFEST) is
 /usr/local _but_ tar tf of the unxz'd tarball is _not_ /usr/local?
 (e.g. pkg_install style) or is this the new way going forward?

No -- it always wants a fully qualified path there.

 So just to give you a better idea of how we manage packages here.
 
 We've realized that if you want to package for FreeBSD (in 9 and
 older), you could get by alright if you knew how to create a
 +CONTENTS file from scratch. For RedHat, it's the SPECFILE. And now
 for FreeBSD 10, it looks an awful lot like a SPECFILE (they are both
 in-fact YAML).
 
 So rather than teach the people here how to use tools, I taught them
 what I think is more important... how to manage +CONTENTS, SPECFILE,
 and now up-and-coming, +MANIFEST.
 
 However, I'd be lying if I said I taught them *all* the gory details.
 In reality, for +CONTENTS they edit a PLIST which is essentially
 +CONTENTS without the @comment MD5: entries. For SPECFILE, they
 manage the full thing except there's a small section that is the
 standard RPM bootstrap that is labeled as do not touch.
 
 From what you posted of +COMPACT_MANIFEST was nice, except it lacks
 the list of files.

There's actually two lists: files and directories.  The principle
difference being that the files list contains SHA256 checksums of each
file.  (Although the files list does also contain sym-links etc, which
don't have a checksum).

There are other data in the full manifest but not in the compact one --
pre- and post- installation or deinstallation scripts are the most
important bits not yet mentioned.

 The basic principle here is that if you have to maintain a list of
 files, and that list ends up being compiled into a +MANIFEST, why not
 just teach your peers how to read/write/maintain +MANIFEST files (in
 version control of course) so that there are never any mysteries as
 to how the package performs.
 
 I know this sounds screwy... but (as with our other YAML files), it's
 really nice because (as is the case with SPECFILE), we
 version-control things as-close to the end-product as possible (take
 for example the case of pre-install or post-install et cetera
 scripts -- I'm sure a tool like pkg create or pkg_create or other
 could do a good job of assembling the pieces into the +MANIFEST, but
 then it's not being version-controlled as closely.

Actually, 'pkg create' basically takes the manifest and uses it to build
the package.  It has a special mode to combine data as presented during
ports compilation into a manifest -- again, this is a compatibility
thing.  If the ports didn't have to support pkg_tools still we could
generate package metadata more directly in the format needed by the
manifest.

 Before this workflow, mistakes were rampant and there wasn't much
 hope of wrangling the oops, forgot to touch the PLIST or oops,
 forgot to update the post-install script mistakes.
 
 I see a nice bright future in re-working my pkgbase to be able to
 supplant metadata into a revision-controlled +MANIFEST.
 
 Ideally, I don't want them to have to be burdened with maintaining
 both a +MANIFEST and +COMPACT_MANIFEST, so it seems most logical to
 generate one from the other (the latter being the revision-controlled
 copy sans-meta-data; the meta-data is added at make time before
 then running validations and generating the tarball).

Internally, the +COMPACT_MANIFEST is generated from the +MANIFEST --
probably best to think in terms of dividing the manifest into sections,
some of which (lists of files, install scripts) your dev people would
actively modify, some of which can be automatically generated by pkg(8)
(shlib provides/requires tracking[*]) and the rest as static boilerplate
you can just paste together as required to create a manifest.

[*] Actually, this is automatically generated at the time the package is
assembled into a tarball.  You don't need to fill in that section of the
manifest eplicitly.

 The problem of breakage to the system by API changes is less
 important, because we track security releng branches and use binaries
 so changes are very slow to creep in. But since *can* use a different
 framework for each/every major release branch, we can track API
 changes quite easily.

Ah.  Remember that pkg(8) isn't part of the base system and is developed
on a timetable independent of the timetable used by FreeBSD itself.
When we release a new version of pkg, it would be applied simultaneously
on all branches 

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Garrett Wollman
In article 20130714064601$3...@grapevine.csail.mit.edu,
dte...@freebsd.org writes:

 [I wrote:]
 It accesses the sqlite database in /var/db/pkg that was previously
 retrieved from the remote repository.

Now from what you explained of pkg, I'm worried that for bsdconfig:

1. Browse packages (nothing else)
2. Exit bsdconfig

and now because the pkg rquery has staged a db, future pkg commands
are now influenced.

Only if you update /usr/local/etc/pkg.conf to set a permanent
PACKAGESITE.  Otherwise, it will notice that the remote repo catalog
you have isn't for your currently selected remote repo, and use that
instead.

 You really shouldn't know about the actual format of the tarballs;
 your time would be better spent learning the new pkg create, pkg
 register, and pkg repo commands.  Depending on your needs, you
 might want to write to the libpkg API instead.

That won't work for us.

You're not going to like the answer, but it does mean that things like
pkg create, pkg register and pkg repo won't work for us.

Congratulations for building your entire workflow around a horrible
kluge straight out of 1993.  FreeBSD, however, is moving on.  (And
it's long past time!)  Your developers will just have to deal.  Or you
can maintain the old cruft for your business -- just don't expect
anyone else to use it, or even want to.

-GAWollman
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Teske, Devin

On Jul 14, 2013, at 8:01 AM, Chris Rees wrote:

 On 14 Jul 2013, at 08:29, Teske, Devin wrote:
 
 To give you an idea as to just how helpful this is...
 
 Imagine the following hierarchy:
 
 src/pkgbase/depend/mystuff/script1
 src/pkgbase/depend/mystuff/textfile1
 src/pkgbase/depend/mystuff/sourcefile.c
 src/pkgbase/depend/mystuff/Makefile
 
 You are a developer. You want to ship a package that contains script1, 
 textfile1, and binary1 (which is compiled by saying make to turn 
 sourcefile.c into binary1)
 
 You want to ship 8 types of packages:
 
 FreeBSD-4.11
 FreeBSD-8.1 (i386)
 FreeBSD-8.1 (amd64)
 RedHat EL 4
 RedHat EL 6 (i386)
 RedHat EL 6 (x86_64)
 Debian Wheezy
 Debian Wheezy 64-bit
 
 This is where my framework comes in-handy...
 
 cd ~/src/pkgbase/freebsd/RELENG_4/category/mystuff
 make
 # it pulled the necessary bits from src/pkgbase/depend/mystuff and built 
 the .tgz
 
 cd ~/src/pkgbase/freebsd/RELENG_8/category/mystuff
 make
 # it pulled the necessary bits from the depend dir and built .tbz
 
 cd ~/src/pkgbase/redhat/rhel4/category/sub-category/mystuff
 make
 # pulled in depend and made .rpm
 
 cd ~/src/pkgbase/redhat/rhel6/category/sub-category/mystuff
 make
 # pulled in depend and made .rpm
 
 etc.
 
 Of course, *any* time the depend tree has binaries in it... you have to 
 first do a make in there on the platform you want to ship the binary for, 
 and then do make depend in the platform-specific tree to pull in the 
 binaries. Once you've done that, you don't have to muck with the depend tree 
 again unless there are changes there.
 
 So, I assume that your prejudice remarks are because you haven't either seen 
 (a) such a platform or (b) such a need for said platform.
 
 Yeah, I could rewrite the freebsd-specific logic to use pkg create, but 
 let me tell you...
 
 When you have to touch a file that needs to get shipped out to multiple 
 platforms...
 
 It's damned nice to be able to build the FreeBSD packages under RedHat 
 *BECAUSE* the redhat RPMs can't be built under anything else (building an 
 RPM on FreeBSD and attempting to install it on RedHat results in an error 
 message similar to this is an rpm for FreeBSD; go away).
 
 Whereas FreeBSD will never balk about a package built on another platform.
 
 It's a huge time-saving measure... not having to jump over to each/every 
 unique platform to package things up *IF/WHEN* you know that there are no 
 binaries in the package *or* you've already checked the pre-compiled 
 binaries into the arch-specific hierarchy.
 
 
 
 
 Or you
 can maintain the old cruft for your business -- just don't expect
 anyone else to use it, or even want to.
 
 
 
 I have no intention of making old-world packages... but I also have no 
 intention of using pkg create.
 
 You still haven't really explained at all why you can't use libpkg.  If it 
 doesn't run on Debian (not tried), it's got to be easier to port it than 
 rewrite a hacked version, hasn't it???  At least then you'll also be 
 contributing back.
 

Simple, really.

Let's take RPM for example. The RPM package format has been ported to other 
platforms.

But, I can't take archivers/rpm4 and build on RPM on FreeBSD and install it on 
RedHat.

This is because the RPM format records the platform that you build your RPM 
on (not the binaries, just the RPM) *into* said RPM.

This actually adds a requirement to the RPM production that the RPMs be 
produced on the platform that they will be installed-to.

Currently, no such restriction exists for the building of FreeBSD packages 
(within our system). This would have been true if we had ported pkg_create (and 
may continue to be true if we ported pkg and its ilk), but let's say for the 
sake of argument that the future of pkg looks bright and it gets ported to 
all sorts of systems (ported in a fashion similar to RPM) *and* we find one day 
that the +MANIFEST starts containing a target-platform (resulting in refusal to 
install a *.txz package because it was rolled on a different platform.

In that case, we'd then prefer to by-pass the tools and use our own method of 
creating the tar-ball to lift such a restriction.

ASIDE: If I knew how to force rpmbuild into creating androgynous packages for 
other architectures, I'd be doing that to life the restriction there too, but I 
haven't figured out that.

Basically... within our pkgbase tree, we like the branch within the tree to 
dictate how a package is built... not what platform you're on. The goal being 
that we can run a single package-build host that builds all of our packages 
from a single platform.
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and 

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Adrian Chadd
... I bet you could do that. I bet you could build the rpm inside a
linux jail and have the relevant uname bits overridden in the right
way.


-adrian

On 14 July 2013 09:52, Teske, Devin devin.te...@fisglobal.com wrote:

 On Jul 14, 2013, at 8:01 AM, Chris Rees wrote:

 On 14 Jul 2013, at 08:29, Teske, Devin wrote:

 To give you an idea as to just how helpful this is...

 Imagine the following hierarchy:

 src/pkgbase/depend/mystuff/script1
 src/pkgbase/depend/mystuff/textfile1
 src/pkgbase/depend/mystuff/sourcefile.c
 src/pkgbase/depend/mystuff/Makefile

 You are a developer. You want to ship a package that contains script1, 
 textfile1, and binary1 (which is compiled by saying make to turn 
 sourcefile.c into binary1)

 You want to ship 8 types of packages:

 FreeBSD-4.11
 FreeBSD-8.1 (i386)
 FreeBSD-8.1 (amd64)
 RedHat EL 4
 RedHat EL 6 (i386)
 RedHat EL 6 (x86_64)
 Debian Wheezy
 Debian Wheezy 64-bit

 This is where my framework comes in-handy...

 cd ~/src/pkgbase/freebsd/RELENG_4/category/mystuff
 make
 # it pulled the necessary bits from src/pkgbase/depend/mystuff and built 
 the .tgz

 cd ~/src/pkgbase/freebsd/RELENG_8/category/mystuff
 make
 # it pulled the necessary bits from the depend dir and built .tbz

 cd ~/src/pkgbase/redhat/rhel4/category/sub-category/mystuff
 make
 # pulled in depend and made .rpm

 cd ~/src/pkgbase/redhat/rhel6/category/sub-category/mystuff
 make
 # pulled in depend and made .rpm

 etc.

 Of course, *any* time the depend tree has binaries in it... you have to 
 first do a make in there on the platform you want to ship the binary for, 
 and then do make depend in the platform-specific tree to pull in the 
 binaries. Once you've done that, you don't have to muck with the depend 
 tree again unless there are changes there.

 So, I assume that your prejudice remarks are because you haven't either 
 seen (a) such a platform or (b) such a need for said platform.

 Yeah, I could rewrite the freebsd-specific logic to use pkg create, but 
 let me tell you...

 When you have to touch a file that needs to get shipped out to multiple 
 platforms...

 It's damned nice to be able to build the FreeBSD packages under RedHat 
 *BECAUSE* the redhat RPMs can't be built under anything else (building an 
 RPM on FreeBSD and attempting to install it on RedHat results in an error 
 message similar to this is an rpm for FreeBSD; go away).

 Whereas FreeBSD will never balk about a package built on another platform.

 It's a huge time-saving measure... not having to jump over to each/every 
 unique platform to package things up *IF/WHEN* you know that there are no 
 binaries in the package *or* you've already checked the pre-compiled 
 binaries into the arch-specific hierarchy.




 Or you
 can maintain the old cruft for your business -- just don't expect
 anyone else to use it, or even want to.



 I have no intention of making old-world packages... but I also have no 
 intention of using pkg create.

 You still haven't really explained at all why you can't use libpkg.  If it 
 doesn't run on Debian (not tried), it's got to be easier to port it than 
 rewrite a hacked version, hasn't it???  At least then you'll also be 
 contributing back.


 Simple, really.

 Let's take RPM for example. The RPM package format has been ported to other 
 platforms.

 But, I can't take archivers/rpm4 and build on RPM on FreeBSD and install it 
 on RedHat.

 This is because the RPM format records the platform that you build your RPM 
 on (not the binaries, just the RPM) *into* said RPM.

 This actually adds a requirement to the RPM production that the RPMs be 
 produced on the platform that they will be installed-to.

 Currently, no such restriction exists for the building of FreeBSD packages 
 (within our system). This would have been true if we had ported pkg_create 
 (and may continue to be true if we ported pkg and its ilk), but let's say for 
 the sake of argument that the future of pkg looks bright and it gets ported 
 to all sorts of systems (ported in a fashion similar to RPM) *and* we find 
 one day that the +MANIFEST starts containing a target-platform (resulting in 
 refusal to install a *.txz package because it was rolled on a different 
 platform.

 In that case, we'd then prefer to by-pass the tools and use our own method of 
 creating the tar-ball to lift such a restriction.

 ASIDE: If I knew how to force rpmbuild into creating androgynous packages for 
 other architectures, I'd be doing that to life the restriction there too, but 
 I haven't figured out that.

 Basically... within our pkgbase tree, we like the branch within the tree to 
 dictate how a package is built... not what platform you're on. The goal being 
 that we can run a single package-build host that builds all of our packages 
 from a single platform.
 --
 Devin

 _
 The information contained in this message is proprietary and/or confidential. 
 If you are not the intended recipient, please: (i) delete the message and 

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Chris Rees
On 14 Jul 2013, at 08:29, Teske, Devin wrote:
 
 To give you an idea as to just how helpful this is...
 
 Imagine the following hierarchy:
 
 src/pkgbase/depend/mystuff/script1
 src/pkgbase/depend/mystuff/textfile1
 src/pkgbase/depend/mystuff/sourcefile.c
 src/pkgbase/depend/mystuff/Makefile
 
 You are a developer. You want to ship a package that contains script1, 
 textfile1, and binary1 (which is compiled by saying make to turn 
 sourcefile.c into binary1)
 
 You want to ship 8 types of packages:
 
 FreeBSD-4.11
 FreeBSD-8.1 (i386)
 FreeBSD-8.1 (amd64)
 RedHat EL 4
 RedHat EL 6 (i386)
 RedHat EL 6 (x86_64)
 Debian Wheezy
 Debian Wheezy 64-bit
 
 This is where my framework comes in-handy...
 
 cd ~/src/pkgbase/freebsd/RELENG_4/category/mystuff
 make
 # it pulled the necessary bits from src/pkgbase/depend/mystuff and built 
 the .tgz
 
 cd ~/src/pkgbase/freebsd/RELENG_8/category/mystuff
 make
 # it pulled the necessary bits from the depend dir and built .tbz
 
 cd ~/src/pkgbase/redhat/rhel4/category/sub-category/mystuff
 make
 # pulled in depend and made .rpm
 
 cd ~/src/pkgbase/redhat/rhel6/category/sub-category/mystuff
 make
 # pulled in depend and made .rpm
 
 etc.
 
 Of course, *any* time the depend tree has binaries in it... you have to first 
 do a make in there on the platform you want to ship the binary for, and then 
 do make depend in the platform-specific tree to pull in the binaries. Once 
 you've done that, you don't have to muck with the depend tree again unless 
 there are changes there.
 
 So, I assume that your prejudice remarks are because you haven't either seen 
 (a) such a platform or (b) such a need for said platform.
 
 Yeah, I could rewrite the freebsd-specific logic to use pkg create, but let 
 me tell you...
 
 When you have to touch a file that needs to get shipped out to multiple 
 platforms...
 
 It's damned nice to be able to build the FreeBSD packages under RedHat 
 *BECAUSE* the redhat RPMs can't be built under anything else (building an RPM 
 on FreeBSD and attempting to install it on RedHat results in an error message 
 similar to this is an rpm for FreeBSD; go away).
 
 Whereas FreeBSD will never balk about a package built on another platform.
 
 It's a huge time-saving measure... not having to jump over to each/every 
 unique platform to package things up *IF/WHEN* you know that there are no 
 binaries in the package *or* you've already checked the pre-compiled binaries 
 into the arch-specific hierarchy.
 
 
 
 
 Or you
 can maintain the old cruft for your business -- just don't expect
 anyone else to use it, or even want to.
 
 
 
 I have no intention of making old-world packages... but I also have no 
 intention of using pkg create.

You still haven't really explained at all why you can't use libpkg.  If it 
doesn't run on Debian (not tried), it's got to be easier to port it than 
rewrite a hacked version, hasn't it???  At least then you'll also be 
contributing back.

Chris


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Craig Rodrigues
On Sat, Jul 13, 2013 at 10:54 PM, Teske, Devin devin.te...@fisglobal.comwrote:



 I assume that poudiere builds packages from ports.


Yes.



  That's not how we build package repositories here (and would expect that
 there are many more like us).


How do you build packages if you are not using FreeBSD ports?  Do you have
your own Makefiles which resemble FreeBSD ports,
or are you doing something completely different?



  We expect to start with an empty directory, go grab packages that we
 want (one by one) from FTP. Then put those tarballs into a directory.
 There's no additional step because we download the INDEX file to the
 repository too. Local repository is built.


What FTP site are you grabbing packages from, the FreeBSD ftp site (or
mirrors) or one of your own?
How are you generating the INDEX file?  Do you append to the INDEX file for
every package which you sucessfully download from FTP,
or are you downloading a previously generated INDEX file?  What does your
local repository of packages consist of if you are
not using FreeBSD ports?





  But I want bsdconfig to work with local repositories without having
 poudriere.


You don't have to use poudriere to build a package repository.  It is just
an optional tool that
is there for you to try out, and it works quite well when building a package
repository based off of FreeBSD ports.  If you don't want to use it, you
don't have to.





  I think so too. But right now a *lot* of unanswered questions.


Possibly, but I think that you are supporting a workflow that people on
this mailing list
don't fully understand.   If you could send out a separate e-mail
describing exactly what your workflow is, step-by-step, that might help you
get better answers
to your questions.   The FreeBSD ports team has been migrating to pkgng
for the past few years, so a lot of people understand the FreeBSD
ports/packages workflow.
because it has been discussed for a few years on multiple mailing lists, and
at various BSD conferences like BSDCan.

Some of the confusion with others on this list is that you are working on
a tool called bsdconfig and committing it into FreeBSD-CURRENT,
but the workflow you are trying to support is slightly different than what
the FreeBSD ports/packages teams have been supporting and adapting
their tools and workflow towards.

I think that if you better describe your workflow in a separate e-mail,
then folks can better recommend how to adapt bsdconfig
to the new pkg tools.  You may need to be open to using
things like libpkg inside bsdconfig, if that is the best solution for
bsdconfig inside FreeBSD.

I agree with you that having a pkgng transition document on the wiki
would be useful for systems integrators.  I know that many
people build products based on FreeBSD and leverage the pkg_XXX tools
for building their own products.  I have done this myself in the past.
Clarifying the transition path for system integrators would be very helpful.
Maybe you can help write the transition doc, since you have the perspective?

--
Craig
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Rainer Duffner

Am 15.07.2013 um 00:43 schrieb Craig Rodrigues rodr...@freebsd.org:

 On Sat, Jul 13, 2013 at 10:54 PM, Teske, Devin 
 devin.te...@fisglobal.comwrote:
 
 
 
 I assume that poudiere builds packages from ports.
 
 
 Yes.
 
 
 
 That's not how we build package repositories here (and would expect that
 there are many more like us).
 
 
 How do you build packages if you are not using FreeBSD ports?  



If I understand him correctly, he uses packages built elsewhere and re-packages 
them into his own format?
http://druidbsd.sourceforge.net/download.shtml#pkgbase

I don't really see any value in this. At some point, his scripts etc. have to 
be packaged anyway?
It should not be a problem running one server/VM for each system.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Teske, Devin

On Jul 14, 2013, at 11:50 AM, Adrian Chadd wrote:

 ... I bet you could do that. I bet you could build the rpm inside a
 linux jail and have the relevant uname bits overridden in the right
 way.
 

There's an idea.
-- 
Devin


 On 14 July 2013 09:52, Teske, Devin devin.te...@fisglobal.com wrote:
 
 On Jul 14, 2013, at 8:01 AM, Chris Rees wrote:
 
 On 14 Jul 2013, at 08:29, Teske, Devin wrote:
 
 To give you an idea as to just how helpful this is...
 
 Imagine the following hierarchy:
 
 src/pkgbase/depend/mystuff/script1
 src/pkgbase/depend/mystuff/textfile1
 src/pkgbase/depend/mystuff/sourcefile.c
 src/pkgbase/depend/mystuff/Makefile
 
 You are a developer. You want to ship a package that contains script1, 
 textfile1, and binary1 (which is compiled by saying make to turn 
 sourcefile.c into binary1)
 
 You want to ship 8 types of packages:
 
 FreeBSD-4.11
 FreeBSD-8.1 (i386)
 FreeBSD-8.1 (amd64)
 RedHat EL 4
 RedHat EL 6 (i386)
 RedHat EL 6 (x86_64)
 Debian Wheezy
 Debian Wheezy 64-bit
 
 This is where my framework comes in-handy...
 
 cd ~/src/pkgbase/freebsd/RELENG_4/category/mystuff
 make
 # it pulled the necessary bits from src/pkgbase/depend/mystuff and built 
 the .tgz
 
 cd ~/src/pkgbase/freebsd/RELENG_8/category/mystuff
 make
 # it pulled the necessary bits from the depend dir and built .tbz
 
 cd ~/src/pkgbase/redhat/rhel4/category/sub-category/mystuff
 make
 # pulled in depend and made .rpm
 
 cd ~/src/pkgbase/redhat/rhel6/category/sub-category/mystuff
 make
 # pulled in depend and made .rpm
 
 etc.
 
 Of course, *any* time the depend tree has binaries in it... you have to 
 first do a make in there on the platform you want to ship the binary for, 
 and then do make depend in the platform-specific tree to pull in the 
 binaries. Once you've done that, you don't have to muck with the depend 
 tree again unless there are changes there.
 
 So, I assume that your prejudice remarks are because you haven't either 
 seen (a) such a platform or (b) such a need for said platform.
 
 Yeah, I could rewrite the freebsd-specific logic to use pkg create, but 
 let me tell you...
 
 When you have to touch a file that needs to get shipped out to multiple 
 platforms...
 
 It's damned nice to be able to build the FreeBSD packages under RedHat 
 *BECAUSE* the redhat RPMs can't be built under anything else (building an 
 RPM on FreeBSD and attempting to install it on RedHat results in an error 
 message similar to this is an rpm for FreeBSD; go away).
 
 Whereas FreeBSD will never balk about a package built on another platform.
 
 It's a huge time-saving measure... not having to jump over to each/every 
 unique platform to package things up *IF/WHEN* you know that there are no 
 binaries in the package *or* you've already checked the pre-compiled 
 binaries into the arch-specific hierarchy.
 
 
 
 
 Or you
 can maintain the old cruft for your business -- just don't expect
 anyone else to use it, or even want to.
 
 
 
 I have no intention of making old-world packages... but I also have no 
 intention of using pkg create.
 
 You still haven't really explained at all why you can't use libpkg.  If it 
 doesn't run on Debian (not tried), it's got to be easier to port it than 
 rewrite a hacked version, hasn't it???  At least then you'll also be 
 contributing back.
 
 
 Simple, really.
 
 Let's take RPM for example. The RPM package format has been ported to other 
 platforms.
 
 But, I can't take archivers/rpm4 and build on RPM on FreeBSD and install it 
 on RedHat.
 
 This is because the RPM format records the platform that you build your 
 RPM on (not the binaries, just the RPM) *into* said RPM.
 
 This actually adds a requirement to the RPM production that the RPMs be 
 produced on the platform that they will be installed-to.
 
 Currently, no such restriction exists for the building of FreeBSD packages 
 (within our system). This would have been true if we had ported pkg_create 
 (and may continue to be true if we ported pkg and its ilk), but let's say 
 for the sake of argument that the future of pkg looks bright and it gets 
 ported to all sorts of systems (ported in a fashion similar to RPM) *and* we 
 find one day that the +MANIFEST starts containing a target-platform 
 (resulting in refusal to install a *.txz package because it was rolled on a 
 different platform.
 
 In that case, we'd then prefer to by-pass the tools and use our own method 
 of creating the tar-ball to lift such a restriction.
 
 ASIDE: If I knew how to force rpmbuild into creating androgynous packages 
 for other architectures, I'd be doing that to life the restriction there 
 too, but I haven't figured out that.
 
 Basically... within our pkgbase tree, we like the branch within the tree 
 to dictate how a package is built... not what platform you're on. The goal 
 being that we can run a single package-build host that builds all of our 
 packages from a single platform.
 --
 Devin
 
 _
 The information contained in this 

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Adrian Chadd
Guys,

Devin runs a _lot_ of FreeBSD stuff at his work. If anything we as a
community should be making his life easier, not act like he's just
clueless and doing it wrong.


-adrian

On 14 July 2013 16:24, Rainer Duffner rai...@ultra-secure.de wrote:

 Am 15.07.2013 um 00:43 schrieb Craig Rodrigues rodr...@freebsd.org:

 On Sat, Jul 13, 2013 at 10:54 PM, Teske, Devin 
 devin.te...@fisglobal.comwrote:



 I assume that poudiere builds packages from ports.


 Yes.



 That's not how we build package repositories here (and would expect that
 there are many more like us).


 How do you build packages if you are not using FreeBSD ports?



 If I understand him correctly, he uses packages built elsewhere and 
 re-packages them into his own format?
 http://druidbsd.sourceforge.net/download.shtml#pkgbase

 I don't really see any value in this. At some point, his scripts etc. have to 
 be packaged anyway?
 It should not be a problem running one server/VM for each system.
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Rainer Duffner

Am 15.07.2013 um 03:15 schrieb Adrian Chadd adr...@freebsd.org:

 Guys,
 
 Devin runs a _lot_ of FreeBSD stuff at his work.


Doubtlessly.
It wouldn't make sense on a small scale.


I assume, his system pre-dates most of the stuff nowadays filed under the 
dev-ops moniker (chef, puppet…).
Probably also has (had) less overhead ;-)



___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-14 Thread Teske, Devin

On Jul 14, 2013, at 3:43 PM, Craig Rodrigues wrote:

On Sat, Jul 13, 2013 at 10:54 PM, Teske, Devin 
devin.te...@fisglobal.commailto:devin.te...@fisglobal.com wrote:


I assume that poudiere builds packages from ports.

Yes.


That's not how we build package repositories here (and would expect that there 
are many more like us).

How do you build packages if you are not using FreeBSD ports?

Two spectrums...

When developer A says he needs xerces, I go to:

(for an active release, such as 8.4 or 9.1)
ftp://ftp.freebsd.org/pub/FreeBSD/releases/{arch}/X.Y-RELEASE/packages/Allftp://ftp.freebsd.org/pub/FreeBSD/releases/%7Barch%7D/X.Y-RELEASE/packages/All

(for an in-active release, such as 8.1 or 9.0)
ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/{arch}/X.Y-RELEASE/packages/Allftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/%7Barch%7D/X.Y-RELEASE/packages/All

Meanwhile, when either developer B or integrator A says I have software X that 
I want to package, we use pkgbase to build the FreeBSD package, the RedHat 
RPM, etc.


  Do you have your own Makefiles which resemble FreeBSD ports,
or are you doing something completely different?


That's essentially what pkgbase is. Except instead of building makefiles that 
build the 3rd-party software, we rely on the 3rd-party compilation process to 
produce the binaries and have a configuration file (pkgbase.conf) which 
describes the vendor layout.

The vendor layout can be anything and do anything and prepare anything. But the 
pkgbase.conf says fileA in the vendor layout goes to src/fileA (where 
src/ looks exactly like the un-packed tar-ball, *minus* +CONTENTS -- which is 
generated from PLIST when you say make).

Since we're using CVS, people that are familiar with CVS would think it's quite 
cute that it does a cvs admin -ko on files that are pulled from the vendor 
branch (resulting in keyword expansion to retain its vendor location -- so if a 
problem happens in the field, we know that the file that should be edited -- 
based on its keyword expansion -- is over in the vendor depend tree which, 
upon modification, will cause a cascading rebuild of packages for releases we 
actively maintain [through a sliding window]).

So it's very similar to ports, but we need the mechanism of being able to (for 
the benefit of developer A/B and integrator A above) freeze the software 
distribution of software X.

The ports tree manages that through a collection of centrally located distfiles 
(which acts as a fall-back for when vendor distfiles dry up). But that method 
doesn't work so well for us because when we find a bug we often fix it 
ourselves in our own local branch and then communicate the fix back the vendor 
(leaving it up to them to incorporate it). The next time around on our 
migration to a new release, *if* we have made a local patch, producing a custom 
version, *and* the vendor has not taken a patch, we'll _first_ re-evaluate the 
new version with a documented test-case or (if that fails) we'll keep the patch 
in-place.

The ports tree moves too fast for us. And many times we find a bug, we end up 
performing cost/benefit analysis only to find that we can live with the bug 
*if* the problem has been fixed already. In rare cases, we'll merge a single 
patch from a future revision to get things going. But we *won't* just blindly 
take a new version simply because it fixes a bug (as it may introduce new bugs 
-- unwinding many man-hours of many testers and developers attempting to lead 
up to a release of a wholly contained full-working software entity).

I hope this helps to explain a little more about how we take FreeBSD releases, 
vet them, and them blast them out to thousands of machines over many customers 
after many months of testing.

Now to say that we don't use ports would be a lie. Sometimes a developer asks 
for a package that is not available on the release we're shipping. This 
happened recently. So I went into the ports tree, built the package on 8.1 
(which had 99% binary packages from 8.1-RELEASE -- see above URLs) and then I 
said make package and threw it in the repository.

ASIDE: For diligence, make describe iirc generates my INDEX entry to append.




We expect to start with an empty directory, go grab packages that we want (one 
by one) from FTP. Then put those tarballs into a directory. There's no 
additional step because we download the INDEX file to the repository too. Local 
repository is built.

What FTP site are you grabbing packages from, the FreeBSD ftp site (or mirrors) 
or one of your own?

We grab from either ftp.f.o or ftp-archive.f.o and plop into an NFS-based repo 
(which has a structure mirroring that of the ftp{,-archive}.f.o structure).



How are you generating the INDEX file?

Well, since we prefer to ship 100% with the packages that were cut with 
X.Y-RELEASE, the INDEX file that's up on the ftp.f.o site does fine (except 
when we have to inject things from ports -- for example, when 

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Baptiste Daroussin
On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote:
 On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:
 
  Hi,
  
  I have just committed (r253305) a change the make pkg_install not being 
  built
  and installed by default on HEAD.
  
  If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
  src.conf(5)
 
 I think while a good move, we need to allow a window of development to 
 re-work other HEAD components.
 
 It might also be worth developing a lint-tool to make sure we get every last 
 instance (both from C code and sh code) within our base before we motion to 
 cut a release with this change.
 
 I for one am effected and will have to change things.

If you are referring to bsdconfig's package management, it is not working anyway
HEAD as we do not and will not provides any pkg_install for HEAD via any of the
usual distribution process: http, ftp, CD.

Secondly if I'm properly reading src.conf.5 bsdconfig is not installed by
default, which means we have a non default code depending on a now non default
code, I don't see a problem here, should I?

regards,
Bapt


pgppDchDwdxvw.pgp
Description: PGP signature


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Mark Linimon
fwiw, nanobsd also is still on pkg_*, but I intend to come up with
patches for that if someone else has not already done so.

mcl
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Baptiste Daroussin
On Sat, Jul 13, 2013 at 06:11:11AM -0500, Mark Linimon wrote:
 fwiw, nanobsd also is still on pkg_*, but I intend to come up with
 patches for that if someone else has not already done so.
 
 mcl

There are lots of patches available everyone to have nanobsd using pkgng

I didn't commit any, because I do find the nanobsd dev a bit weird, lots of
different version everywhere, lots of different patches etc.

regards,
Bapt


pgplUJmU6pUFf.pgp
Description: PGP signature


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Outback Dingo
On Sat, Jul 13, 2013 at 7:59 AM, Baptiste Daroussin b...@freebsd.orgwrote:

 On Sat, Jul 13, 2013 at 06:11:11AM -0500, Mark Linimon wrote:
  fwiw, nanobsd also is still on pkg_*, but I intend to come up with
  patches for that if someone else has not already done so.
 
  mcl

 There are lots of patches available everyone to have nanobsd using pkgng

 I didn't commit any, because I do find the nanobsd dev a bit weird, lots of
 different version everywhere, lots of different patches etc.


would someone like to forward their patches for pkg on nanobsd to the list


 regards,
 Bapt

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Teske, Devin

On Jul 13, 2013, at 1:07 AM, Baptiste Daroussin wrote:

On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote:
On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:

Hi,

I have just committed (r253305) a change the make pkg_install not being built
and installed by default on HEAD.

If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
src.conf(5)

I think while a good move, we need to allow a window of development to re-work 
other HEAD components.

It might also be worth developing a lint-tool to make sure we get every last 
instance (both from C code and sh code) within our base before we motion to cut 
a release with this change.

I for one am effected and will have to change things.

If you are referring to bsdconfig's package management,

If you choose to read into it that way, sure. Yes. that's what I'm talking 
about.

NOTE: I can't be the only person in HEAD that's using pkg_install still.



it is not working anyway
HEAD as we do not and will not provides any pkg_install for HEAD via any of the
usual distribution process: http, ftp, CD.


I'm quite aware that changes will be needed. I'm still interested in a 
transition wiki if one exists.




Secondly if I'm properly reading src.conf.5

If there's a reference to WITH_BSDCONFIG in src.conf.5, that needs to be 
removed. I wasn't aware that it was referenced in there.


bsdconfig is not installed by
default,

Wrong, please see...
http://svnweb.freebsd.org/base?view=revisionrevision=252862



which means we have a non default code depending on a now non default
code,

Wrong.


I don't see a problem here, should I?


Yes.
--
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Teske, Devin

On Jul 13, 2013, at 8:17 AM, Teske, Devin wrote:


On Jul 13, 2013, at 1:07 AM, Baptiste Daroussin wrote:

On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote:
On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:

Hi,

I have just committed (r253305) a change the make pkg_install not being built
and installed by default on HEAD.

If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
src.conf(5)

[snip]

I for one am effected and will have to change things.

If you are referring to bsdconfig's package management,

[snip] Yes. that's what I'm talking about. [snip]


it is not working anyway
HEAD as we do not and will not provides any pkg_install for HEAD via any of the
usual distribution process: http, ftp, CD.



The FTP mirror of packages is going away? (if you said yes and pointed at a 
prior conversation about leading up to this, I would not be surprised -- I'm 
just questioning it because I don't see the value in pairing-down methods of 
acquisition)

If this is the case, what's the surviving acquisition method? A custom TCP 
protocol perhaps?

There may be those that wish to use pkg in the pkg_add manner and download 
things and then inspect them manually before adding them. For example, I often 
go the freshports.orghttp://freshports.org to find a package that fills a 
need... download it from the official FTP server(s), inspect all of them, and 
choose the one that best fits me need (and only then installing from the local 
file; tossing the rest). If I go through the pkg tool, I have to inspect 
things *after* they've been installed which is not to my satisfaction.


[snip
bsdconfig is not installed by
default,

Wrong, please see...
http://svnweb.freebsd.org/base?view=revisionrevision=252862
[snip]

The first thing that comes to mind in reprogramming bsdconfig's package 
management for pkgng...

We have a *very* large list of FTP mirrors. This will presumably be replaced 
with a list of pkg mirrors.

Do we have such a list that we want to program into the base configuration of 
bsdconfig?
--
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Kimmo Paasiala
On Sat, Jul 13, 2013 at 7:46 PM, Teske, Devin devin.te...@fisglobal.com wrote:

 On Jul 13, 2013, at 8:17 AM, Teske, Devin wrote:


 On Jul 13, 2013, at 1:07 AM, Baptiste Daroussin wrote:

 On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote:
 On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:

 Hi,

 I have just committed (r253305) a change the make pkg_install not being built
 and installed by default on HEAD.

 If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
 src.conf(5)

 [snip]

 I for one am effected and will have to change things.

 If you are referring to bsdconfig's package management,

 [snip] Yes. that's what I'm talking about. [snip]


 it is not working anyway
 HEAD as we do not and will not provides any pkg_install for HEAD via any of 
 the
 usual distribution process: http, ftp, CD.



 The FTP mirror of packages is going away? (if you said yes and pointed at a 
 prior conversation about leading up to this, I would not be surprised -- I'm 
 just questioning it because I don't see the value in pairing-down methods of 
 acquisition)

 If this is the case, what's the surviving acquisition method? A custom TCP 
 protocol perhaps?

 There may be those that wish to use pkg in the pkg_add manner and download 
 things and then inspect them manually before adding them. For example, I 
 often go the freshports.orghttp://freshports.org to find a package that 
 fills a need... download it from the official FTP server(s), inspect all of 
 them, and choose the one that best fits me need (and only then installing 
 from the local file; tossing the rest). If I go through the pkg tool, I 
 have to inspect things *after* they've been installed which is not to my 
 satisfaction.


 [snip
 bsdconfig is not installed by
 default,

 Wrong, please see...
 http://svnweb.freebsd.org/base?view=revisionrevision=252862
 [snip]

 The first thing that comes to mind in reprogramming bsdconfig's package 
 management for pkgng...

 We have a *very* large list of FTP mirrors. This will presumably be replaced 
 with a list of pkg mirrors.

 Do we have such a list that we want to program into the base configuration of 
 bsdconfig?
 --
 Devin


Come on, this only concerns 10-CURRENT. Where is it stated that the
FTP mirrors for FreeBSD 8/9 packages are going away?

-Kimmo
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Teske, Devin

On Jul 13, 2013, at 11:16 AM, Kimmo Paasiala wrote:

 On Sat, Jul 13, 2013 at 7:46 PM, Teske, Devin devin.te...@fisglobal.com 
 wrote:
 
 On Jul 13, 2013, at 8:17 AM, Teske, Devin wrote:
 
 
 On Jul 13, 2013, at 1:07 AM, Baptiste Daroussin wrote:
 
 On Fri, Jul 12, 2013 at 11:52:19PM +, Teske, Devin wrote:
 On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:
 
 Hi,
 
 I have just committed (r253305) a change the make pkg_install not being built
 and installed by default on HEAD.
 
 If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
 src.conf(5)
 
 [snip]
 
 I for one am effected and will have to change things.
 
 If you are referring to bsdconfig's package management,
 
 [snip] Yes. that's what I'm talking about. [snip]
 
 
 it is not working anyway
 HEAD as we do not and will not provides any pkg_install for HEAD via any of 
 the
 usual distribution process: http, ftp, CD.
 
 
 
 The FTP mirror of packages is going away? (if you said yes and pointed at a 
 prior conversation about leading up to this, I would not be surprised -- I'm 
 just questioning it because I don't see the value in pairing-down methods of 
 acquisition)
 
 If this is the case, what's the surviving acquisition method? A custom TCP 
 protocol perhaps?
 
 There may be those that wish to use pkg in the pkg_add manner and download 
 things and then inspect them manually before adding them. For example, I 
 often go the 
 freshports.orghttps://urldefense.proofpoint.com/v1/url?u=http://freshports.org/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=Ft5J2W3Nm8xze74zARHsiglLFTGOYrxAzaubbP7wvcM%3D%0As=cb1df85e237d1a6be19f981e8b352419dd056ea0f4e782b330cb9c7cfd15fda5
  to find a package that fills a need... download it from the official FTP 
 server(s), inspect all of them, and choose the one that best fits me need 
 (and only then installing from the local file; tossing the rest). If I go 
 through the pkg tool, I have to inspect things *after* they've been 
 installed which is not to my satisfaction.
 
 
 [snip
 bsdconfig is not installed by
 default,
 
 Wrong, please see...
 http://svnweb.freebsd.org/base?view=revisionrevision=252862
 [snip]
 
 The first thing that comes to mind in reprogramming bsdconfig's package 
 management for pkgng...
 
 We have a *very* large list of FTP mirrors. This will presumably be replaced 
 with a list of pkg mirrors.
 
 Do we have such a list that we want to program into the base configuration 
 of bsdconfig?
 --
 Devin
 
 
 Come on, this only concerns 10-CURRENT.

Right; which is why -current is CC'd.


 Where is it stated that the
 FTP mirrors for FreeBSD 8/9 packages are going away?
 

I don't know where you got the impression that I was concerned with 8/9 
packages.

I'm strictly concerned with HEAD and strictly concerned with planning for the 
future.

So yes... I'm asking... in a HEAD world, what is the officially supported 
method of acquisition?

I saw mention on a page that rsync access will provided for those who want to 
mirror but my I'm not really interested in mirroring while I would like to 
continue to be able to grab packages myself without installing them.

Maybe the answer is that pkg has a method of acquiring a single package 
(without dependencies) without downloading it. That would solve my personal 
workflow issue (again, I like to download tarballs, inspect them, and then 
install them from the locally downloaded file -- then if it passes validation 
tests, that downloaded file is migrated to our own distribution servers; it's a 
work-flow for validating packages -- it has less to do with how pkgng works and 
more just whether I can get packages in a fashion such as FTP or HTTP or 
whatever.

Now that aside, bsdconfig currently is a different story. To rewrite the 
packages module to work in a HEAD world for HEAD, using pkgng servers, I'm 
going to have to change the way things are done. Currently there is an 
abstraction layer for fetching packages from media (which can be FTP, HTTP, 
HTTP Proxy, NFS, Local Directory, CDROM, a DOS partition, a UFS partition, and 
[god forbid] Floppy -- with perhaps SMB/CIFS down the road). When it wants to 
add a package, it calls the routine to get the package data by name on 
standard-out and pumps it into pkg_add (after of course initializing the media).

This abstraction layer is useful to pkgng as it prepares the source. What I 
*suspect* will have to stop happening is the direct-fetching of the package 
data and piping that into a program (however, I *could* continue to do that .. 
pkg supports adding of local packages and iirc supports reading from 
standard-in). Instead, perhaps just call pkg in a small-handful of different 
ways (pointed at an ftp server; pointed at a local file; pointed at http; 
etc.). But the preparation of an NFS mount would be handled by the abstraction 
layer leading up to the calling of pkg.

That issue aside (whether to have pkg do the bidding or to use the 

Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Mark Felder
On Sat, Jul 13, 2013, at 13:54, Teske, Devin wrote:
 
 
 If FTP access (or any of the other remote access methods) are going away
 for HEAD pkg access, I'll need to know so I can make the appropriate
 changes in the HEAD branch of bsdconfig.


It's simpler than you think. The new pkg uses libfetch. You can have
your PACAKGESITE be file:// ftp:// http:// https:// ... 

I do suspect that HTTP_PROXY support is probably not available as I
recall seeing a post where someone was asking about that getting
implemented for fetch. I'll have to research that again, though.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Teske, Devin

On Jul 13, 2013, at 4:51 PM, Mark Felder wrote:

 On Sat, Jul 13, 2013, at 13:54, Teske, Devin wrote:
 
 
 If FTP access (or any of the other remote access methods) are going away
 for HEAD pkg access, I'll need to know so I can make the appropriate
 changes in the HEAD branch of bsdconfig.
 
 
 It's simpler than you think. The new pkg uses libfetch. You can have
 your PACAKGESITE be file:// ftp:// http:// https:// ... 
 

In bsdconfig (as one might expect), before a user installs a package, he or she 
is presented with a list of packages with descriptions (and perhaps other 
limited information).

ASIDE: For efficiency, I will actually need three things: (1) a list of all 
packages (2) their descriptions and (3) their run-time dependencies.

NOTE: I'll need run-time dependencies so that as they're checking boxes in the 
UI, I can update accordingly (has nothing to do with how the dependencies are 
handled when the packages get installed; I'll let pkg handle that when it comes 
time, but for the UI and for the review screen, I want the user to be 
informed and I can do that more efficiently if I have a master-list and don't 
have to make continuous queries).

That being said, I was planning on doing a pkg rquery to get that master list 
of [minimally] 3-pieces of information.

What protocol does that rquery run on? What would one have to do to set up 
their own server that can accept an rquery? (our customers don't have 
Internet access)

Last but not least...

Can you do an rquery on a local repository? (say, one that has been mounted 
via NFS or some other media, local or otherwise but looking like a local 
filesystem once-mounted). What would be required to get a local repository like 
that going?



 I do suspect that HTTP_PROXY support is probably not available as I
 recall seeing a post where someone was asking about that getting
 implemented for fetch. I'll have to research that again, though.

Thanks. Keep me up to date on that.

Neither sysinstall nor bsdinstall really give HTTP_PROXY access much thought 
(they support it, but only minimally). They just construct a raw HTTP header 
with the FTP URL and send that directly to the proxy. One cute thing it does 
(when initializing the connection) is test for Squid and if-so, appends 
;type=i to the URL (to force binary download versus ascii).

No support for proxy-server authentication (however user/pass authentication 
for the FTP server is passed through to the proxy-server).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Craig Rodrigues
On Sat, Jul 13, 2013 at 8:18 PM, Teske, Devin devin.te...@fisglobal.comwrote:


 Can you do an rquery on a local repository? (say, one that has been
 mounted via NFS or some other media, local or otherwise but looking like a
 local filesystem once-mounted). What would be required to get a local
 repository like that going?


(1)  Install the port:  ports-mgmgt/poudriere , and follow the instructions
at: https://fossil.etoilebsd.net/poudriere/
(2)  Build your own repository of packages
(3)  Make that repository available in a local file system or NFS

I believe that pkg and poudriere address pretty much all the questions
you have been asking,
so you just need to set up the environment so that you can do experiments
to integrate it with bsdconfig.

PC-BSD and DragonFlyBSD have switched to using pkg, so it is quite well
tested.

Baptiste and the rest of the pkgng and poudriere team have put a lot of
effort and thought into this,
and I believe you will be pleased with the results once you migrate over to
these tools.

--
Craig
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Peter Wemm
On Sat, Jul 13, 2013 at 11:54 AM, Teske, Devin
devin.te...@fisglobal.com wrote:

 So yes... I'm asking... in a HEAD world, what is the officially supported 
 method of acquisition?

This has been answered elsewhere, but to be absolutely clear:

* 10.0 will ship with pkgng format packages.

* pkgng will be configured to pull via http from project
infrastructure by default.

* The project will not be providing pkg_install format packages via
any method for 10.x.

This doesn't stop third parties building and providing them
unofficially but my understanding is that they won't be on any project
operated freebsd.org sites.

Footnote... *.cc.freebsd.org are generally project operated sites.
-- 
Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV
UTF-8: So you can \342\200\231 .. for when a ' just won't do
brueffer ZFS must be the bacon of file systems. everything's better with ZFS
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Peter Wemm
On Sat, Jul 13, 2013 at 9:24 PM, Peter Wemm pe...@wemm.org wrote:

 Footnote... *.cc.freebsd.org are generally project operated sites.

.. gah.. generally *NOT* project operated sites.  They're third party
/ regional / local.

-- 
Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Teske, Devin

On Jul 13, 2013, at 9:24 PM, Peter Wemm wrote:

 On Sat, Jul 13, 2013 at 11:54 AM, Teske, Devin
 devin.te...@fisglobal.com wrote:
 
 So yes... I'm asking... in a HEAD world, what is the officially supported 
 method of acquisition?
 
 This has been answered elsewhere, but to be absolutely clear:
 
 * 10.0 will ship with pkgng format packages.
 

Ok. Respectfully not what I was inquiring about.


 * pkgng will be configured to pull via http from project
 infrastructure by default.
 

Thanks. Good to know. So when you don't define PACKAGESITE, it defaults to HTTP.

How about rquery? What protocol does that use? and what does it talk to?



 * The project will not be providing pkg_install format packages via
 any method for 10.x.
 

Yes, I know this from previous announcements (not the topic at-hand; 
respectfully).



 This doesn't stop third parties building and providing them
 unofficially but my understanding is that they won't be on any project
 operated freebsd.org sites.
 

Question: Where can I learn more about the actual format of what's in the new 
tarballs? This is going to be important not for bsdconfig, but $work (we have 
our own build platform; I'm going to have to rewrite it from mastering PLIST 
files to mastering YAML MANIFEST files and I want to know all the gritty 
details).
-- 
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-13 Thread Teske, Devin

On Jul 13, 2013, at 8:33 PM, Craig Rodrigues wrote:

On Sat, Jul 13, 2013 at 8:18 PM, Teske, Devin 
devin.te...@fisglobal.commailto:devin.te...@fisglobal.com wrote:

Can you do an rquery on a local repository? (say, one that has been mounted 
via NFS or some other media, local or otherwise but looking like a local 
filesystem once-mounted). What would be required to get a local repository like 
that going?


(1)  Install the port:  ports-mgmgt/poudriere , and follow the instructions at: 
https://fossil.etoilebsd.net/poudriere/https://urldefense.proofpoint.com/v1/url?u=https://fossil.etoilebsd.net/poudriere/k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0Ar=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0Am=ITD4dgEgVeGBJaxdFd9siyBYqr8XS3YLpKM1XB2BIlo%3D%0As=3f2fd1a60ccad617509daca0f40579fc7aa17cfc3a6ee176d38a4f2de3c913a0
(2)  Build your own repository of packages

I assume that poudiere builds packages from ports.

That's not how we build package repositories here (and would expect that there 
are many more like us).

We expect to start with an empty directory, go grab packages that we want (one 
by one) from FTP. Then put those tarballs into a directory. There's no 
additional step because we download the INDEX file to the repository too. Local 
repository is built.

Is it reasonable to assume that this is still possible? We don't want to have 
to set up poudriere or build our own packages for packages that were already 
built at the time of release. (we don't track anything except security releng 
branches).



(3)  Make that repository available in a local file system or NFS


That's what we want.


I believe that pkg and poudriere address pretty much all the questions you 
have been asking,
so you just need to set up the environment so that you can do experiments to 
integrate it with bsdconfig.


But I want bsdconfig to work with local repositories without having poudriere.



PC-BSD and DragonFlyBSD have switched to using pkg, so it is quite well 
tested.


Cool.


Baptiste and the rest of the pkgng and poudriere team have put a lot of effort 
and thought into this,
and I believe you will be pleased with the results once you migrate over to 
these tools.


I think so too. But right now a *lot* of unanswered questions.
--
Devin

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] No more pkg_install on HEAD by default

2013-07-12 Thread Teske, Devin
On Jul 12, 2013, at 4:16 PM, Baptiste Daroussin wrote:

 Hi,
 
 I have just committed (r253305) a change the make pkg_install not being built
 and installed by default on HEAD.
 
 If you are still relying on it, be careful and add WITH_PKGTOOLS=yes in your
 src.conf(5)

I think while a good move, we need to allow a window of development to re-work 
other HEAD components.

It might also be worth developing a lint-tool to make sure we get every last 
instance (both from C code and sh code) within our base before we motion to cut 
a release with this change.

I for one am effected and will have to change things.

Can you point us at a guide (or even better, a Wiki) that can smooth the 
process?
-- 
Devin

NOTE: I cut ports off the CC as they don't need to worry about this (this 
being heading toward cut of a RELEASE with untold uses of pkg_tools in base)

_
The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org