Re: Ideas and questions on pkgsrc
Justin C. Sherrill wrote: that appears to be removable for the same reasons (overkill). Sascha I think was looking at a BSD-based replacement for groff, or maybe I'm conflating it with the mandoc work. I think someone was looking at heirloom troff at somepoint - but I think that ran into incompatibilities with drift in the -man macros since 4.4BSD, etc. anyhow - not really relavent, just for archive sake
Re: Ideas and questions on pkgsrc
On Mon, 19 Apr 2010, Chris Turner wrote: Justin C. Sherrill wrote: that appears to be removable for the same reasons (overkill). Sascha I think was looking at a BSD-based replacement for groff, or maybe I'm conflating it with the mandoc work. I think someone was looking at heirloom troff at somepoint - but I think that ran into incompatibilities with drift in the -man macros since 4.4BSD, etc. anyhow - not really relavent, just for archive sake See replacements for man parts: http://mdocml.bsd.lv/ I see it is in NetBSD, DragonFly, and OpenBSD now. And in FreeBSD ports. (You already know about this part.) Also Thorsten at MirBSD has maintained the historical (non-c++, non-GNU) nroff (with its additional text formatting macros and tools) since 2003. http://www.mirbsd.org/htman/i386/man1/nroff.htm http://www.mirbsd.org/htman/i386/man7/mandoc.htm http://www.mirbsd.org/htman/i386/man7/ms.htm etc ... That could be considered to replace groff. And if anyone needs groff for some special project then they can use it from pkgsrc.
Re: Ideas and questions on pkgsrc
Chris Turner wrote: Justin C. Sherrill wrote: - General ideas about the bulk builds and binary installs; I've been staring at it so long I can't see the forest because there's all these trees in the way. Yeah.. Lots of these ideas crossing my mind today as well - this DNS-in-base thing kind of spurs a lot of thinking about direction (or maybe thats my life right now :) aaanyhow.. some things come to mind: 1) I've got some 'pkgsrc profile' scripts lying around. They are basically lists of package names, which use 'rcorder' to create / install a heirarchy of packages.. e.g. just to pick a random example, your 'trac' profile might pull in your 'apache' profile, and also your 'python' and 'developer tools' profile. I'm in the process of dusting off code / revamping my website - I'll try to get that released somehow - maybe getting it into pkgsrc/pkgutils might make sense - and providing a few stock ones of these might help.. Well, w/o having seen the code, this sounds like a bit of a hack :) Also I'm not sure what problem you're solving. Pkgsrc already has working package dependencies. The serious issue is with handling upgrades. which brings to mind: 2) nrelease vs pkgsrc vs contrib interfacing - If the plan is to modularize alot of contrib, it probably makes sense to get some kind of toolkit in place for this. I for one like having some things 'always built', and running nrelease every now and again has always seemed a bit like something that I do as an afterthought - and having this be a more 'usual' thing, might make the release go more smoothly / etc. e.g. if my 'make buildworld' would build dfly world, a pkgsrc bootstrap, and some core tools, like say BIND :) - that would be pretty hip - and if 'make installworld' would drop those into place without blowing up my base install, that would be hip too. Eh, to do this properly you'd have to upgrade all packages depending on your 'base' packages and all packages depending on those packages and so on (transitive closure). Keeping stuff in base has the disadvantage that pkgsrc doesn't really know when they get upgraded, so if you upgrade something in base to an incompatible version things will break unless you manually upgrade every pkgsrc package depending on that (directly or indirectly). Nevermind that pkgsrc might not yet have a newer version that will work with the thing you upgraded. I'd say that is a strong argument for a slim base (or making sure /usr/pkg is almost completely independent of base). Aggelos
Re: Ideas and questions on pkgsrc
Justin C. Sherrill wrote: So, after seeing that PostgresQL is moving services from FreeBSD to Debian because of ease of packaging, and seeing Ivan Voras's idea for a stable branch of ports similar to the quarterly pkgsrc releases, I've been thinking about the pkgsrc service. (Here's the links for the aforementioned items:) http://blog.hagander.net/archives/167-PostgreSQL-infrastructure-updates.html http://ivoras.sharanet.org/blog/tree/2010-04-11.of-ports-and-men.html Here's my thoughts. Platform viability is being determined mostly by how well it handles the software installation and availability. Ports has historically been a big reason for FreeBSD, and I'd argue that the best thing in Debian/Ubuntu is apt-get. PC-BSD has rolled their own packaging method for similar reasons. So, it stands to reason that the easier it is to use pkgsrc on DragonFly, the better off DragonFly will be. I've seen a number of new people come into #dragonflybsd on EFNet recently asking questions about their first install, and one of the first answers is usually pkg_radd. I love that we have such an easy tool for quick installs. I and probably a good chunk of the people reading this are now used to pkgsrc and DragonFly and can deal with various issues, but I worry that being used to it makes it harder for us to see/fix these problems. I'm looking for some ideas or suggestions on what we could do. Well, I for one can't get used to some of the pkgsrc issues (see below). I've been thinking - - A first-time guide on the website and maybe the LiveDVD desktop that talks about common packages to install via pkg_radd so people don't have to guess names. Directing them towards pkg_search should help, no? - Are there other contrib/ items we can move to pkgsrc? Aside from cvs, I don't see any obvious candidates. We need a lot of stuff for our build, other stuff carries local patches that are hard to get upstream, other stuff (nvi, less, file) is very useful to have and are not really a burden. We should eventually move sendmail out of base (setting up dma by default), but I'm not sure how easy that is and what's the sendmail status in pkgsrc. - Would it be useful to see all the bulk build reports from i386/x86_64/2.6/2.7 that I build? It might, so sure. - Would it be worth having a separate mailing list for those reports? They can be pretty frequent - several per day. Dunno. - Should we explore putting more onto the LiveDVD? Maybe. What would be the use case scenario? My by far most important gripe w/ pkgsrc is the inability to do mass upgrades from binary packages in a straightforward manner. Not even sure if it's anything the pkgsrc developers are concerned with. Things I could use help with: - The crontab output from the bulk builds will list the new packages from each build. It's rsync output, so anything with a - : net/foo-1.2 - net/foo-1.2. If someone can come up with a script that parses out those lines from emails and can create an email or webpage showing what's updated for which arch, that would really help people. What would you want that for? To the admin, having a way to ask For which of my installed packages are there newer versions that I can upgrade too sounds more useful. - General ideas about the bulk builds and binary installs; I've been staring at it so long I can't see the forest because there's all these trees in the way. Like I have asked in the past, NOT having incomplete (while packages are uploading) package sets be visible would be very useful. At the very least we should have a way to detect that case manually. I don't think we can solve all the problems on our own. I bet NetBSD people are coming up against some of them too. Anybody know what their policies are and what, if anything, is coming down the pipeline as a solution? Aggelos
Re: Ideas and questions on pkgsrc
Aggelos Economopoulos wrote: Well, w/o having seen the code, this sounds like a bit of a hack :) Also I'm not sure what problem you're solving. Pkgsrc already has working package dependencies. The serious issue is with handling upgrades. yup. possibly so. Problem solving: installing a box that does X, and specifying a box that does 'X' simply. e.g.: my desktop is 80% windowmaker nautilus firefox aterm emacs and webdev is 80% apache postgres p-languages therefore 'workstation' == require desktop require webdev and I'm done - and I don't care if package 'foo' switched to latest-desktop-technology-blah v4.6d in the meantime, and I don't have to track it in the makefiles. also makes it easy to make tools that automate system builds, etc.. Eh, to do this properly you'd have to upgrade all packages depending on your 'base' packages and all packages depending on those packages and so on (transitive closure). I'd only really do this for 'core' things - like a DNS server, MTA, etc. (those pesky types of things that keep getting moved out of base) these tend to be highly contested leaf-nodes, which don't really have dependancies - aka things that 'used to be' part of the OS. I personally blow away the entire system at each release - and don't really do much in the middle (effectively 'upgrading all packages') . Test box gets test packages - if I get the time to do it. Though I'm adressing my workflow as I get back up to speed from where I was ~2y ago..
Re: Ideas and questions on pkgsrc
Aggelos Economopoulos wrote: My by far most important gripe w/ pkgsrc is the inability to do mass upgrades from binary packages in a straightforward manner. Not even sure if it's anything the pkgsrc developers are concerned with. Can't you do this if you have the right make targets (e.g. check binary packages first) and pkg_rolling-replace? I think the problem you mean is more is 'mass upgrades from binary packages in a remote location' .. I think yeah - a lot of pkgsrc folks have a site-local /usr/pkgsrc/packages hanging out in their automounter somewhere ..
Re: Ideas and questions on pkgsrc
Chris Turner wrote: Aggelos Economopoulos wrote: Well, w/o having seen the code, this sounds like a bit of a hack :) Also I'm not sure what problem you're solving. Pkgsrc already has working package dependencies. The serious issue is with handling upgrades. yup. possibly so. Problem solving: installing a box that does X, and specifying a box that does 'X' simply. e.g.: my desktop is 80% windowmaker nautilus firefox aterm emacs *Shrug*. OK, sure, why not. But for general use, metapackages seem to work well enough. In fact I'm not sure why you can't just reuse *that* concept (i.e. create appropriate metapackages instead of ad-hoc scripts. and webdev is 80% apache postgres p-languages Heh. Those are all arbitrary choices really. Not terribly useful to a general user audience. therefore 'workstation' == require desktop require webdev and I'm done - and I don't care if package 'foo' switched to latest-desktop-technology-blah v4.6d in the meantime, and I don't have to track it in the makefiles. You have to track it in your 'profile' though, not sure why that's better. also makes it easy to make tools that automate system builds, etc.. Eh, to do this properly you'd have to upgrade all packages depending on your 'base' packages and all packages depending on those packages and so on (transitive closure). I'd only really do this for 'core' things - like a DNS server, MTA, etc. (those pesky types of things that keep getting moved out of base) these tend to be highly contested leaf-nodes, which don't really have dependancies - aka things that 'used to be' part of the OS. I personally blow away the entire system at each release Yah, see, that's what Justin was talking about. Tell *that* to someone who's been using apt-get/yum/urpmi/whatever and they'll just feel sorry for you. - and don't really do much in the middle (effectively 'upgrading all packages') . Test box gets test packages - if I get the time to do it. Though I'm adressing my workflow as I get back up to speed from where I was ~2y ago.. Aggelos
Re: Ideas and questions on pkgsrc
Chris Turner wrote: Aggelos Economopoulos wrote: My by far most important gripe w/ pkgsrc is the inability to do mass upgrades from binary packages in a straightforward manner. Not even sure if it's anything the pkgsrc developers are concerned with. Can't you do this if you have the right make targets (e.g. check binary packages first) and pkg_rolling-replace? Notice 'straightforward'. Consider also 'reliable'. The competition has 'single command' also. I think the problem you mean is more is 'mass upgrades from binary packages in a remote location' .. Yes. I think yeah - a lot of pkgsrc folks have a site-local /usr/pkgsrc/packages hanging out in their automounter somewhere .. How would that help w/ my problem? Let me mention here that I appreciate pkgsrc and I think it's the only viable solution for DragonFly for the forseeable future. But it is still not where I'd want it to be. Aggelos
Re: Ideas and questions on pkgsrc
Aggelos Economopoulos wrote: Heh. Those are all arbitrary choices really. Not terribly useful to a general user audience. s/above/gnome-and-3d-compositing-windowmanager/ and as for tracking, no - I just track the leaf nodes. intermediate cruft stays buried in the makefile deps. Yah, see, that's what Justin was talking about. Tell *that* to someone who's been using apt-get/yum/urpmi/whatever and they'll just feel sorry for you. and I for them. - C
Re: Ideas and questions on pkgsrc
On 4/13/2010 3:33 AM, Chris Turner wrote: Can't you do this if you have the right make targets (e.g. check binary packages first) and pkg_rolling-replace? I don't think anyone yet has a proper tool or script to do this reliably. A while back, when I tried one suggested with rolling-replace in the dragonfly-digest noted from pkgsrc mails, which probably just used the source packages. It became obvious to me it wasn't tested before being recommended, nor end user friendly and reliable on DragonFly, to deal with software updating issues. I ended up with a hosed system. I wouldn't mind this solution, if it was easy to setup, safe, and effective. I think the problem you mean is more is 'mass upgrades from binary packages in a remote location' .. I'd expect an average user or sysadmin would prefer the speed and simplicity of this, if you keep the last known working binaries online when newer versions are broken. I wouldn't mind seeing a publicly posted link to the pkgsrc mk.conf file used to build the binary packages, so they can be duplicated easily and adjusted locally as the sysadmin sees fit.
Re: Ideas and questions on pkgsrc
On Tue, April 13, 2010 11:49 am, Ed Berger wrote: I don't think anyone yet has a proper tool or script to do this reliably. A while back, when I tried one suggested with rolling-replace in the dragonfly-digest noted from pkgsrc mails, which probably just used the source packages. It became obvious to me it wasn't tested before being recommended, nor end user friendly and reliable on DragonFly, to deal with software updating issues. I ended up with a hosed system. I wouldn't mind this solution, if it was easy to setup, safe, and effective. I could have sworn I did it and it worked... Maybe retroactive wishful thinking on my part? In any case, I know I've used pkg_rolling-replace from source with success. I'd expect an average user or sysadmin would prefer the speed and simplicity of this, if you keep the last known working binaries online when newer versions are broken. I wouldn't mind seeing a publicly posted link to the pkgsrc mk.conf file used to build the binary packages, so they can be duplicated easily and adjusted locally as the sysadmin sees fit. http://gitweb.dragonflybsd.org/~justin/simplepbulk.git/blob_plain/HEAD:/mk-base.conf I've been unsetting PKG_DEVELOPER since that recently changed to have unprivileged builds, but that shouldn't affect anyone not doing a bulk build. The only useful part would be PKG_DEFAULT_OPTIONS=dri inet6
Re: Ideas and questions on pkgsrc
On Tue, 13 Apr 2010, Aggelos Economopoulos wrote: talks about common packages to install via pkg_radd so people don't have to guess names. Directing them towards pkg_search should help, no? pkgin search My by far most important gripe w/ pkgsrc is the inability to do mass upgrades from binary packages in a straightforward manner. Not even sure if it's anything the pkgsrc developers are concerned with. pkgin upgrade What would you want that for? To the admin, having a way to ask For which of my installed packages are there newer versions that I can upgrade too sounds more useful. pkgin list If pkgin doesn't work well for you, please report the problems.
Re: Ideas and questions on pkgsrc
:On Tue, 13 Apr 2010, Aggelos Economopoulos wrote: : : talks about common packages to install via pkg_radd so people don't have : to guess names. : : Directing them towards pkg_search should help, no? : :pkgin search : : My by far most important gripe w/ pkgsrc is the inability to do mass : upgrades from binary packages in a straightforward manner. Not even sure : if it's anything the pkgsrc developers are concerned with. : :pkgin upgrade : : What would you want that for? To the admin, having a way to ask For : which of my installed packages are there newer versions that I can : upgrade too sounds more useful. : :pkgin list : :If pkgin doesn't work well for you, please report the problems. I haven't yet tried pkgin. But here's my question: One thing apt-get does very well is it doesn't blow up your existing installs if it can't completely update everything that needs to be updated to install the application you want. That is, it stages everything and makes sure it has all the pieces before it touches the installed base. Does pkgin do that? -Matt Matthew Dillon dil...@backplane.com
Re: Ideas and questions on pkgsrc
On Tue, 13 Apr 2010, Matthew Dillon wrote: I haven't yet tried pkgin. But here's my question: One thing apt-get does very well is it doesn't blow up your existing installs if it can't completely update everything that needs to be updated to install the application you want. That is, it stages everything and makes sure it has all the pieces before it touches the installed base. Does pkgin do that? It should. It use the pkg_summary database to make sure dependencies are correct, there aren't conflicts, there is enough space to download and install, and that shared libraries are provided. Then it downloads all dependencies before it begins. Note it needs the packages' metadata built with recording this shared libraries provided or required if that is desired. pkgin has had some problems for me, but it has also been a good timesaver. Probably with more use and more feedback, it can be improved. As for Debian, if you have any details on the stages you can point me to that would be great. (I do use apt-get but it has been a while since I studied it.)
Re: Ideas and questions on pkgsrc
On Tue, April 13, 2010 3:06 am, Aggelos Economopoulos wrote: Aside from cvs, I don't see any obvious candidates. We need a lot of stuff for our build, other stuff carries local patches that are hard to get upstream, other stuff (nvi, less, file) is very useful to have and are not really a burden. We should eventually move sendmail out of base (setting up dma by default), but I'm not sure how easy that is and what's the sendmail status in pkgsrc. Well, that's cvs and sendmail. Looking at /contrib, there's nothing else that appears to be removable for the same reasons (overkill). Sascha I think was looking at a BSD-based replacement for groff, or maybe I'm conflating it with the mandoc work. Even if we just get those parts out, it's still that much less maintenance where we're just treading water instead of doing new stuff. Is anyone working on finishing DMA at this point? What would you want that for? To the admin, having a way to ask For which of my installed packages are there newer versions that I can upgrade too sounds more useful. That would work too, but my goal is to have a public list of what's now available as binary packages. You can see what's in pkgsrc online (pkgsrc.se), and you can run programs like pkg_chk to see what's newer in /usr/pkgsrc, but if we're to get to a good binary installation method, we need to be able to see online what's newer _and_ available as binary.
Re: Ideas and questions on pkgsrc
Justin C. Sherrill wrote: - General ideas about the bulk builds and binary installs; I've been staring at it so long I can't see the forest because there's all these trees in the way. Yeah.. Lots of these ideas crossing my mind today as well - this DNS-in-base thing kind of spurs a lot of thinking about direction (or maybe thats my life right now :) aaanyhow.. some things come to mind: 1) I've got some 'pkgsrc profile' scripts lying around. They are basically lists of package names, which use 'rcorder' to create / install a heirarchy of packages.. e.g. just to pick a random example, your 'trac' profile might pull in your 'apache' profile, and also your 'python' and 'developer tools' profile. I'm in the process of dusting off code / revamping my website - I'll try to get that released somehow - maybe getting it into pkgsrc/pkgutils might make sense - and providing a few stock ones of these might help.. which brings to mind: 2) nrelease vs pkgsrc vs contrib interfacing - If the plan is to modularize alot of contrib, it probably makes sense to get some kind of toolkit in place for this. I for one like having some things 'always built', and running nrelease every now and again has always seemed a bit like something that I do as an afterthought - and having this be a more 'usual' thing, might make the release go more smoothly / etc. e.g. if my 'make buildworld' would build dfly world, a pkgsrc bootstrap, and some core tools, like say BIND :) - that would be pretty hip - and if 'make installworld' would drop those into place without blowing up my base install, that would be hip too. or maybe having a separate target, like 'make buildpkgs' in the master makefile - in order to preserve modularity, I dunno.. Side point: Does the installer work in a vkernel, btw? I've never tried - but this would probably help with release engineering / end user documentation testing.. 3) hack-a-thons - on line to start. I think these would be a great idea. I for one am always scratching my head to catch up with the list, etc.. Perhaps I should be on leaf / IRC more too. my fault there.. but basically, there's lots of stupid little things that I never get around to doing, coz there's always some other thing. But yeah. anyhow.. I spend far too much time writing emails to the list and not writing code! so, with that in mind - I'm off to catch up on git finally, so I can commit my 1 liner from like a frigging month ago. w00t!
