Re: what next for the pkg_install rewrite
On Thu, Sep 2, 2010 at 2:52 AM, Baptiste Daroussin b...@freebsd.org wrote: sqlite if we goes on is BSD-like licensed so it should ne be a problem... ... That would be great, the json lib I use for testing purpose cjson (http://sourceforge.net/projects/cjson/)... Okay, I'll pull both of these in when I get a chance. I might actually be able to use cjson to help with the logging I need to do in the database and in the install phase. How hard would it be to add support for the new manifest on you libpkg which is really great by the way :) After looking at cjson I could probably add support in a couple of hours. Some things in my lib aren't finalized though, so I wouldn't recommend spending time on it right now. How hard would it be to allow other developers to work with you on libpkg? separate the tasks etc. Before and during the summer it would have been pretty simple because I was working on git and hg. At the end of the summer I decided to move over to p4 until I have at least the entire current pkg system supported (and then I'll probably move the project back over to a git repo). Anyone can look at that and submit patches, but I don't want to create a communal repository until I'm finished with the lib. I'm in an... interesting... situation right now, I just moved and my bsd machine has no way of getting on the internet. I do have some pretty big changes in the pipeline that I haven't been able to commit yet, though (namely logging in the database and some naming convention changes), so go ahead and look, but I really wouldn't recommend touching too much for a while because a lot of stuff might change. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On Thu, Sep 2, 2010 at 2:53 AM, Ivan Voras ivo...@freebsd.org wrote: Repository format as in ftp server layout or something else? Eh, format isn't really the word I needed there. I'm not really sure, I guess the whole repository *system*(?) has areas it can improve? One of the ideas I've had is basically what MacPorts does (mentioned by Tim), and another is using an index to resolve the actual location of packages (and possibly give suggestions). I'll put some stuff on the wiki if I get a chance. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
Before and during the summer it would have been pretty simple because I was working on git and hg. At the end of the summer I decided to move over to p4 until I have at least the entire current pkg system supported (and then I'll probably move the project back over to a git repo). I think git is a good choice :) Anyone can look at that and submit patches, but I don't want to create a communal repository until I'm finished with the lib. I'm in an... interesting... situation right now, I just moved and my bsd machine has no way of getting on the internet. I do have some pretty big changes in the pipeline that I haven't been able to commit yet, though (namely logging in the database and some naming convention changes), so go ahead and look, but I really wouldn't recommend touching too much for a while because a lot of stuff might change. That's fair, you did most of the coding work, the fact that you keep the repo as yours is fair, I think we need someone to be the master of the code (don't know how to explain that with other words :), that is important for coherence and consistency. So before contributing code and being able to separate tasks we will have to wait for your bsd machine to get access to the internet :). That would let us the time needed to discuss all the features (and the design) that would become pkg_install2. regards, Bapt ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
Eh, why didn't this thread spring up before summer of code got under way (or during the many weeks it was running)? Concerning the database situation, sqlite would be cool. If we can ship the sqlite code either with libpkg or with the pkg_install tools, it's a win (I don't read licenses, someone will have to tell me if we can do that). Realistically, if this is doable, then there can be no opposition because the pkg_install tools as they are now create their own database, so nothing is changing. Plus we'll get free concurrency, which everybody wants. If I can get an okay on this, I'll start reworking libpkg to handle it. The manifest format really does need to change. We should go with JSON and either write our own specialized parser/writer or go with something like yajl. Writing a parser and a writer that plug into my libpkg will give you conversion tools (both forwards and backwards) for free and adding properties and changing formats isn't really a big deal because I store package information in a plist(5) (from OS X) like structure. Bringing a fat library into base just for manifest parsing will meet with resistance, so if you guys can find some really small json lib (I'm talking 1 or 2 files) that's license compatible, I'll happily suck it into libpkg. The lua stuff is way too far off in dream land for my taste... The things that you guys want to drop from plists are things gcooper has been wanting to get rid of and replace for a while, and libarchive restoring group/mode/owner perms has been the plan for a while now I believe (I actually just ignore this stuff in libpkg). Separating ports and packages is silly, because they need to coexist. Like gcooper pointed out, ports should be using the pkg tools to build and install packages. The repository format is something that you guys didn't really touch on that I think could use some fixing, but I'll wait to see where this all goes and maybe throw some ideas on a wiki page later on. In the mean time, I'll keep building libpkg too work with the pkg system in its current state. Dave. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On Thu, Sep 02, 2010 at 12:34:58AM -0700, David Forsythe wrote: Eh, why didn't this thread spring up before summer of code got under way (or during the many weeks it was running)? I was expecting for your reply :) Sorry I didn't launch this thread before that because I didn't get enough spare time before summer, nor during it. Concerning the database situation, sqlite would be cool. If we can ship the sqlite code either with libpkg or with the pkg_install tools, it's a win (I don't read licenses, someone will have to tell me if we can do that). Realistically, if this is doable, then there can be no opposition because the pkg_install tools as they are now create their own database, so nothing is changing. Plus we'll get free concurrency, which everybody wants. If I can get an okay on this, I'll start reworking libpkg to handle it. sqlite if we goes on is BSD-like licensed so it should ne be a problem, it is designed to be embeded (one .h and .c to compile) no need to import the full sqlite sources. (see www.fossil-scm.org from the author of sqlite). The manifest format really does need to change. We should go with JSON and either write our own specialized parser/writer or go with something like yajl. Writing a parser and a writer that plug into my libpkg will give you conversion tools (both forwards and backwards) for free and adding properties and changing formats isn't really a big deal because I store package information in a plist(5) (from OS X) like structure. Bringing a fat library into base just for manifest parsing will meet with resistance, so if you guys can find some really small json lib (I'm talking 1 or 2 files) that's license compatible, I'll happily suck it into libpkg. That would be great, the json lib I use for testing purpose cjson (http://sourceforge.net/projects/cjson/) which is really simple to use, BSD license and can be embeded. I have writen a pkg_register which is a quick and dirty version of pkg_add -O just to be a dropin replacement of PKG_CMD for ports to register new package with the new manifest (without the need of patching bsd.port.mk) The lua stuff is way too far off in dream land for my taste... I think the only interest of lua in pkg_install would be to allow hooks written in lua but this is not very important in the state we are). So let's forget lua for the moment :) The things that you guys want to drop from plists are things gcooper has been wanting to get rid of and replace for a while, and libarchive restoring group/mode/owner perms has been the plan for a while now I believe (I actually just ignore this stuff in libpkg). Separating ports and packages is silly, because they need to coexist. Like gcooper pointed out, ports should be using the pkg tools to build and install packages. Agree with that The repository format is something that you guys didn't really touch on that I think could use some fixing, but I'll wait to see where this all goes and maybe throw some ideas on a wiki page later on. I expect for your ideas on the wiki page. In the mean time, I'll keep building libpkg too work with the pkg system in its current state. How hard would it be to add support for the new manifest on you libpkg which is really great by the way :) How hard would it be to allow other developers to work with you on libpkg? separate the tasks etc. regards, bapt pgp1yH8ctUmmA.pgp Description: PGP signature
Re: what next for the pkg_install rewrite
On 2 September 2010 09:34, David Forsythe dfors...@freebsd.org wrote: Eh, why didn't this thread spring up before summer of code got under way (or during the many weeks it was running)? Yes, that would have been beneficial. For what it's worth, I've been trying to start it without success for years :) As for your other questions, I think you should read http://wiki.freebsd.org/Pkg_install2_specs - most of them will be answered there (also follow the wiki links). The repository format is something that you guys didn't really touch on that I think could use some fixing, but I'll wait to see where this all goes and maybe throw some ideas on a wiki page later on. Repository format as in ftp server layout or something else? ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On Sep 2, 2010, at 12:34 AM, David Forsythe wrote: In the mean time, I'll keep building libpkg too work with the pkg system in its current state. Any really viable plan to change the manifest structure will have to include a means to continue to support the old structure during the transition period. Tim ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On Sep 2, 2010, at 12:34 AM, David Forsythe wrote: Separating ports and packages is silly, because they need to coexist. Like gcooper pointed out, ports should be using the pkg tools to build and install packages. I've been impressed with how MacPorts handles this. If I understand correctly, you can specify a package you want and the options you want with it and the system will download a pre-built package with those options (if one exists) or pull the source and build the port, all transparently. Even better, the central repository sees all of these requests, so can prioritize which option sets are most popular. Tim ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On Thu 02 Sep 2010 at 08:56:50 PDT Tim Kientzle wrote: On Sep 2, 2010, at 12:34 AM, David Forsythe wrote: Separating ports and packages is silly, because they need to coexist. Like gcooper pointed out, ports should be using the pkg tools to build and install packages. I've been impressed with how MacPorts handles this. If I understand correctly, you can specify a package you want and the options you want with it and the system will download a pre-built package with those options (if one exists) or pull the source and build the port, all transparently. Even better, the central repository sees all of these requests, so can prioritize which option sets are most popular. Would that include any options specified in make.conf? ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
Bapt b...@freebsd.org writes: On Fri, Aug 20, 2010 at 02:09:59PM +0200, Julien Laffaye wrote: On Fri, Aug 20, 2010 at 12:00 PM, Ivan Voras ivo...@freebsd.org wrote: 2. XML is a bad idea. Great in theory, wonderful in my browser, but a bloated plaintext file with a lot of complexity. I would prefer a database solution that could be dumped to a simple text file format if needed. XML, at least in my proposal, would not be used for the system package database, but would replace (either in part or all with a single file) today's + files in the packages, together with other purposes where some metadata transfer is required. That said, I would also be happy with other self-described formats like JSON. XML is the front runner because it's more standard (industry standard, whether someone likes this fact or not!) and there is already a parser in base. The pro of XML here is that we have a parser in base, The con is its verbosity. But anyway the manifest is not meant to be human readable. Nevertheless, I prefer JSON, because it's less complex than XML and we can have a parser/emiter in a single 500SLOC .c file. And it looks sexier than XML, eh : http://pastebin.com/8hzPSSJC +1 for JSON, XML would have been interesting if we add a validating parser in base which would have help us to verify the package integrity, but expat isn't a good validating parser. writting a json parser/emiter is easy, and there already exists tons of BSDL friendly license JSON parser. For exemple we could import yajl (BSDL) into base or use cJSON (MIT) which consist in one simple C + header file We can as well use Lua tables to store package database. Their syntax is close to JSON. Besides, I think it's better to divorce ports from base so that pkg_* tools can evolve faster and are not limited to dependencies from base. The only thing we'd need to leave in base is smth like pkg_bootstrap. IMO, this chicken and egg problem is getting quite annoying. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
2010/8/30 Anonymous swel...@gmail.com: Bapt b...@freebsd.org writes: We can as well use Lua tables to store package database. Their syntax is close to JSON. Besides, I think it's better to divorce ports from base so that pkg_* tools can evolve faster and are not limited to dependencies from base. The only thing we'd need to leave in base is smth like pkg_bootstrap. IMO, this chicken and egg problem is getting quite annoying. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org yes lua could be a good idea for that purpose and it also would allow users to easily extend pkg_install. I have begun that: http://wiki.freebsd.org/Pkg_install2_specs if you could add ideas and all stuff you have in mind about what has to be and what could be in pkg_install, I'll cleanup the page after some time, when all the needed discussion would be done and will transform this page into real specifications. Concerning having pkg_install into base or not, I really have no opinion? regards, Bapt ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On 30 August 2010 09:27, Anonymous swel...@gmail.com wrote: We can as well use Lua tables to store package database. Their syntax is close to JSON. Besides, I think it's better to divorce ports from base so that pkg_* tools can evolve faster and are not limited to dependencies from base. The only thing we'd need to leave in base is smth like pkg_bootstrap. IMO, this chicken and egg problem is getting quite annoying. Speaking of Lua, I had a thread on this in -current which went IMO fairly well, mostly because Lua is a clean and easy language to import compared to, e.g. Perl, TCL or Python. As I see it, there will not be heavy opposition if Lua is to be imported. In short, if there is going to be a scripting language for pkg_*, Lua is sort-of pre-approved - as opposed to ksh and others mentioned here. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On Mon, Aug 30, 2010 at 11:27:58AM +0200, Ivan Voras wrote: On 30 August 2010 09:27, Anonymous swel...@gmail.com wrote: We can as well use Lua tables to store package database. Their syntax is close to JSON. Besides, I think it's better to divorce ports from base so that pkg_* tools can evolve faster and are not limited to dependencies from base. The only thing we'd need to leave in base is smth like pkg_bootstrap. IMO, this chicken and egg problem is getting quite annoying. Speaking of Lua, I had a thread on this in -current which went IMO fairly well, mostly because Lua is a clean and easy language to import compared to, e.g. Perl, TCL or Python. As I see it, there will not be heavy opposition if Lua is to be imported. In short, if there is going to be a scripting language for pkg_*, Lua is sort-of pre-approved - as opposed to ksh and others mentioned here. Lua would make a nice addition for an unrelated reason (I don't follow -current so someone may have mentioned this already): there is an interest in replacing the Forth/FICL pieces of the FreeBSD bootloader with something in Lua instead. Rink Springer and I discussed this (either in Email or on IRC, I forget), and both of us have interest in such. For those curious about Lua, I highly recommend the book Programming in Lua (2nd Edition). -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On 30 August 2010 10:22, Baptiste Daroussin b...@freebsd.org wrote: I have begun that: http://wiki.freebsd.org/Pkg_install2_specs if you could add ideas and all stuff you have in mind about what has to be and what could be in pkg_install, I'll cleanup the page after some time, when all the needed discussion would be done and will transform this page into real specifications. I like the proposal for the manifest format. I've added some small ideas. Concerning having pkg_install into base or not, I really have no opinion? Oh it should be in base, the operating system isn't complete without it. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
I agree having a packaging@ mailing list would help to discuss about pkg_install stuff. We need to summarize the ideas of each one, then discuss about it. Only then we can specified what needs to be done and how (keeping in mind that we need to keep compatibility at least as a fallback or directly). when that part of the work is done we could be able to propose the statement to portmgr for them to validate and to update the ports policy if needed (hopefully they would have taken part in the discussion so the validation should be formal) If everyone agree with the following: What I propose to do is to let the discussion going on for the next couple of days (or moredepending on the activity on this thread) and then write down all that has been proposed (pro, con) then we should be able to start the specification for pkg_install next generation :) I propose myself as someone has to do the job, but if you think you or other are better suited for the job go ahead propose yourself or name the one you want to punish. back to the subject. I personnaly believe that pkg_install needs a complete rewrite. we have strong basis, libarchive, the GSoC code: libpkg, pkg_complete and many more (pkg_patch) (sorry if I forgot some). The new specifications has to be validated by portmgr at the begining of the project and the code should repect that specifications: once we have all accepted what would become the new pkg_install we won't add features or behaviour that are not in the specification until the release and integration in base. We need to centralized the code in the same place with the same scm (I would love if we could avoid p4 :)) ideas are welcomed :) We need to keep the compatiblity (as much as possible) with the existing pkg_install, through wrappers, scripts, or fallback code. The Plist has to be reworked: a new clean format which represents the new features that ports provide I also agree that pkgname and version should be separated A new policy on package names should be written to prevent the apr case or at least internally the package origin should be used to identify the packages. perhaps we could keep the informations on the build options within the metadatas Package should know which architecture they are made for : i386, amd64, noarch, etc. It would allow to prevent rebuilding of cross plateform package on clusters, it would allow to prevent being able to install sparc64 package on i386? regards, Bapt pgpi8FIUGtdlb.pgp Description: PGP signature
Re: what next for the pkg_install rewrite
On 20/08/2010, Garrett Cooper gcoo...@freebsd.org wrote: 1. SQLite was killed before because of complexity and because it was needs another package in base that isn't BSD licensed. That's why And... both ideas are completely wrong. SQLite can be imported as a single C file + header, which you must agree is practically the optimum, and its license is public domain which is, if anything, freer than BSDL and eminently compatible with it. everyone in the know has been pushing for BDB 1.8x (in base). People BDB in base offer exactly one (1) single feature, if you can call it that: storing and retrieving key-value binary blobs. It has no practical concurrency support, it's locking is laughable and upgrading data stored as binary blobs is even more nightmarish than maintaining the current plist format (and it will lead to similar uglyness soon - rather than upgrading the data piece called x, I'm sure developers will introduce new keys called x_extdata, x_moredata etc etc). SQLite on the other hand solves all these in the following way: 1) stores proper records, not blobs 2) has proper locking concurrency support, ACID 3) the database schema can be modified on the fly for upgrading - fields can be added to existing table while preserving data. 4) is endian-agnostic, 32/64-bit agnostic and portable (I mean the database file) to an extremely large number of platforms. It is already used as a system database in OS X, iPhone and Android. Note that I'm promoting SQLite to replace the /var/db/pkg/* tree of directories and files with ad-hoc structure - all data in there should be in one SQLite database, which is stored in a single file. suggested that to me back when I was trying to get off the ground in GSoC 2006, they did it to you before Ivan, and I'm sure they've done it before in the past. We need to implement things in terms of BDB first, but make the APIs generic enough so that it _could_ be extended to SQLite if people chose to do the work later on. Not planning an API for a transactional database in the first place will bring the whole thing down in the long term, and in any case will certainly push things 10 more years in the future. Note that SQLite can *not* be considered a drop-in replacement for BDB (any version of BDB) because in that case you will gain far less than is optimal. If you haven't used SQLite yet, please try it, from C, and then see if you can reevaluate your comment on its complexity. If you don't like SQL, this is also fine. You are not out of the game, there are many other parts to work on. As for backward compatibility: basically it can be done in two ways: 1) build a one-time converter that will read the present plist database and output a new database or 2) build a compatibility layer which would support both the old and the new format at the same time, so people upgrading will continue to use their old data. I consider the latter wasteful in terms of developer resources and because bit-rot will eventually make support for the old format decrepit. 2. XML is a bad idea. Great in theory, wonderful in my browser, but a bloated plaintext file with a lot of complexity. I would prefer a database solution that could be dumped to a simple text file format if needed. XML, at least in my proposal, would not be used for the system package database, but would replace (either in part or all with a single file) today's + files in the packages, together with other purposes where some metadata transfer is required. That said, I would also be happy with other self-described formats like JSON. XML is the front runner because it's more standard (industry standard, whether someone likes this fact or not!) and there is already a parser in base. Again (and I can't overemphasize this): no matter what we do, say, etc, unless we have buy-in from portmgr beforehand on anything here, the work will never see a ports/src CVS/svn repo, etc. This includes work done for past GSoCs, and current GSoCs. With all respect to portmgr, I expect it to keep a cool head and acknowledge the following: 1) Decisions must be made on objective terms which include consideration for clean / elegant implementation, long-term maintainability and standards. If there is to be a rewrite, it must be built to stand the test of next 10 years of usage, and not start compromising even before it is started. 2) Except in extreme cases, portmgr should decide based on functionality and not have that much say in the specifics of the implementation. Basically, the portmgr should in the first place approve the feature spec, and the src people should worry about the details of what can and cannot be done. This is not a src vs portmgr war, this is separation of duty. (of course there are overlaps in people's memberships) ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to
Re: what next for the pkg_install rewrite
On Fri, Aug 20, 2010 at 12:00 PM, Ivan Voras ivo...@freebsd.org wrote: And... both ideas are completely wrong. SQLite can be imported as a single C file + header, which you must agree is practically the optimum, and its license is public domain which is, if anything, freer than BSDL and eminently compatible with it. And if we don't want to build a sqlite.so which can conflict with the version from ports, we can statically link libpkg to it. Incognito. BDB in base offer exactly one (1) single feature, if you can call it that: storing and retrieving key-value binary blobs. It has no practical concurrency support, it's locking is laughable and upgrading data stored as binary blobs is even more nightmarish than maintaining the current plist format (and it will lead to similar uglyness soon - rather than upgrading the data piece called x, I'm sure developers will introduce new keys called x_extdata, x_moredata etc etc). If we are going key-value storage, I think that TinyCDB is worth a look. SQLite on the other hand solves all these in the following way: 1) stores proper records, not blobs 2) has proper locking concurrency support, ACID 3) the database schema can be modified on the fly for upgrading - fields can be added to existing table while preserving data. 4) is endian-agnostic, 32/64-bit agnostic and portable (I mean the database file) to an extremely large number of platforms. It is already used as a system database in OS X, iPhone and Android. Note that I'm promoting SQLite to replace the /var/db/pkg/* tree of directories and files with ad-hoc structure - all data in there should be in one SQLite database, which is stored in a single file. Indeed, all this points can help. The major drawback of SQLite, in my opinion, is the inclusion of long SQL statement in c code. So if we are going this way, we must have strict rules to avoid polluting the source. [...] As for backward compatibility: basically it can be done in two ways: 1) build a one-time converter that will read the present plist database and output a new database or 2) build a compatibility layer which would support both the old and the new format at the same time, so people upgrading will continue to use their old data. I consider the latter wasteful in terms of developer resources and because bit-rot will eventually make support for the old format decrepit. Agreed, a one time converter seems to be the solution. Especially if the transition occur between two major FreeBSD version (say, 9 - 10). 2. XML is a bad idea. Great in theory, wonderful in my browser, but a bloated plaintext file with a lot of complexity. I would prefer a database solution that could be dumped to a simple text file format if needed. XML, at least in my proposal, would not be used for the system package database, but would replace (either in part or all with a single file) today's + files in the packages, together with other purposes where some metadata transfer is required. That said, I would also be happy with other self-described formats like JSON. XML is the front runner because it's more standard (industry standard, whether someone likes this fact or not!) and there is already a parser in base. The pro of XML here is that we have a parser in base, The con is its verbosity. But anyway the manifest is not meant to be human readable. Nevertheless, I prefer JSON, because it's less complex than XML and we can have a parser/emiter in a single 500SLOC .c file. And it looks sexier than XML, eh : http://pastebin.com/8hzPSSJC ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On Fri, Aug 20, 2010 at 02:09:59PM +0200, Julien Laffaye wrote: On Fri, Aug 20, 2010 at 12:00 PM, Ivan Voras ivo...@freebsd.org wrote: And... both ideas are completely wrong. SQLite can be imported as a single C file + header, which you must agree is practically the optimum, and its license is public domain which is, if anything, freer than BSDL and eminently compatible with it. And if we don't want to build a sqlite.so which can conflict with the version from ports, we can statically link libpkg to it. Incognito. BDB in base offer exactly one (1) single feature, if you can call it that: storing and retrieving key-value binary blobs. It has no practical concurrency support, it's locking is laughable and upgrading data stored as binary blobs is even more nightmarish than maintaining the current plist format (and it will lead to similar uglyness soon - rather than upgrading the data piece called x, I'm sure developers will introduce new keys called x_extdata, x_moredata etc etc). If we are going key-value storage, I think that TinyCDB is worth a look. I also think tinycdb is a good candidate: very simple, clean and fast and And as most of the access are readonly, the fact that is a constant database is not a problem, I already tested tinycdb in my freebsd port of pkgin (an apt like for pkgsrc) and result in terms of performance and simplicity were clearly in favor of tinycdb. but if we go to SQLite it is not a problem for me. [...] As for backward compatibility: basically it can be done in two ways: 1) build a one-time converter that will read the present plist database and output a new database or 2) build a compatibility layer which would support both the old and the new format at the same time, so people upgrading will continue to use their old data. I consider the latter wasteful in terms of developer resources and because bit-rot will eventually make support for the old format decrepit. Agreed, a one time converter seems to be the solution. Especially if the transition occur between two major FreeBSD version (say, 9 - 10). I think that for the /var/db/pkg we should make a converter for a one time transition. But tools like pkg_add or pkg_info should be able to works with both new format and old one (at least through a wrapper) 2. XML is a bad idea. Great in theory, wonderful in my browser, but a bloated plaintext file with a lot of complexity. I would prefer a database solution that could be dumped to a simple text file format if needed. XML, at least in my proposal, would not be used for the system package database, but would replace (either in part or all with a single file) today's + files in the packages, together with other purposes where some metadata transfer is required. That said, I would also be happy with other self-described formats like JSON. XML is the front runner because it's more standard (industry standard, whether someone likes this fact or not!) and there is already a parser in base. The pro of XML here is that we have a parser in base, The con is its verbosity. But anyway the manifest is not meant to be human readable. Nevertheless, I prefer JSON, because it's less complex than XML and we can have a parser/emiter in a single 500SLOC .c file. And it looks sexier than XML, eh : http://pastebin.com/8hzPSSJC +1 for JSON, XML would have been interesting if we add a validating parser in base which would have help us to verify the package integrity, but expat isn't a good validating parser. writting a json parser/emiter is easy, and there already exists tons of BSDL friendly license JSON parser. For exemple we could import yajl (BSDL) into base or use cJSON (MIT) which consist in one simple C + header file regards, Bapt pgpHAWbNVzOnv.pgp Description: PGP signature
Re: what next for the pkg_install rewrite
Garrett Cooper yaneurab...@gmail.com writes: The emphasis that Florent made too was to remove crud in pkg_install and libpkg and get things down to more of a library form so we could develop thin wrappers above pkg_install with logical functions (like apt-get, yum, etc does with fetching, whereas rpm does with installation, etc), instead of maintaining pkg_install the way it is today. From perspective of a user I'd like those wrappers be written in a scripted language so they're easy to hack. Because sh(1) doesn't have smth like FFI we'd need one command that exports all functions from the library. Some port managing tools could then bypass those wrappers when doing complex tasks. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On Aug 20, 2010, at 9:27 AM, Anonymous wrote: Garrett Cooper yaneurab...@gmail.com writes: The emphasis that Florent made too was to remove crud in pkg_install and libpkg and get things down to more of a library form so we could develop thin wrappers above pkg_install with logical functions (like apt-get, yum, etc does with fetching, whereas rpm does with installation, etc), instead of maintaining pkg_install the way it is today. From perspective of a user I'd like those wrappers be written in a scripted language so they're easy to hack. Because sh(1) doesn't have smth like FFI we'd need one command that exports all functions from the library. Some port managing tools could then bypass those wrappers when doing complex tasks. As long as we don't get into wrapper script hell (greater than 2 levels deep), sure :).___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On Thu, Aug 19, 2010 at 4:38 PM, Bapt b...@freebsd.org wrote: Hi, Now that GSoC is over a lot of good work has been done on pkg_install. I think it would be great to organize the way the on going work on pkg_install will be done, for that purpose we need someone to officially manage the work, to validate what will be implemented/cleanup, and distribute the work among volunteers. I think this is the best way for a new pkg_install really appear. I'd like to contribute to the work on pkg_install, and would like to help cocordinate the effort to prevent reinventing the wheel, and to be ensure the code is consistent etc. I know there are many like minded people out there who might like to help, please identify yourself, either publicly or privately so we can put together a team to find a pkg_install replacement. Hello, I agree: we need to create a task force to work on pkg_install and thus create a dynamic. The very first step of such a team will be to brainstorm: what do we want to achieve? what can be improved? There are a lot of areas of potential discussions: packing list format, local database format, ... In my opinion, trying to be 100% compatible with the actual tools will slow down the project. I am thinking, for example, about the slave/master modes which made sense when we used a temporary directory, but less if we want to extract the files to their final destination via libarchive. Then, this specification will need to be approved by portmgr@ so the actual coding can start! Best regards, Julien ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On 19/08/2010, Julien Laffaye jlaff...@freebsd.org wrote: There are a lot of areas of potential discussions: packing list format, local database format, ... In my opinion, trying to be 100% compatible with the actual tools will slow down the project. I am thinking, for example, about the slave/master modes which made sense when we used a temporary directory, but less if we want to extract the files to their final destination via libarchive. Then, this specification will need to be approved by portmgr@ so the actual coding can start! Like many people in this discussion I have done some work on pkg_* and for what it's worth, here's what I would like changed: - Fully specify and separate package name from its version - metadata should not record apache-2.2.13 but apache, 2.2.13 to better support upgrading and dependancies. - Debian-like dependancies - the suggests variety, as well as ranged-dependancies - package X depends on Y versions W through Z. - A wrapper for all pkg_ tools to use, implemented with libarchive. This wrapper would allow preparation of the whole archive layout in-memory, together with simulating common file system operations like chmod, chown, rmdir, mkdir, rename, unlink, etc. and would as a last step offer to serialize this virtual file system to an archive. - Policy to forbid the lazy-maintainer dances with package names, such as package names depending on config flags used, etc. - this probably needs more thinking through. Essentially, I want to avoid things like what happened to the apr port - names like apr-ipv6-devrandom-gdbm-db42-1.4.2.9.3.1_1 Of course, this would better be if documented somewhere semi-permanent - in our wiki for example. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On 08/19/2010 12:50, Ivan Voras wrote:
- Fully specify and separate package name from its version - metadata
should not record apache-2.2.13 but apache, 2.2.13 to better
support upgrading and dependancies.
- Debian-like dependancies - the suggests variety, as well as
ranged-dependancies - package X depends on Y versions W through Z.
- A wrapper for all pkg_ tools to use, implemented with libarchive.
This wrapper would allow preparation of the whole archive layout
in-memory, together with simulating common file system operations
like chmod, chown, rmdir, mkdir, rename, unlink, etc. and would as a
last step offer to serialize this virtual file system to an archive.
- Policy to forbid the lazy-maintainer dances with package names, such
as package names depending on config flags used, etc. - this probably
needs more thinking through. Essentially, I want to avoid things like
what happened to the apr port - names like
apr-ipv6-devrandom-gdbm-db42-1.4.2.9.3.1_1
Adding to this I would like to see a central database created for
packages that have been removed like in Slackware Linux. They keep a
file in /var/log/preserved_packages with a flat text format with the
file name looking like:
${PORTNAME}-${PORTVERSION}${PORTREVISION}-`date +%Y%m%d%H%M%S`
With the contents being that of all the actions that were performed
upon package removal and any output from errors etc.
Id like to see that same approach achieved within /var/db/preserved or
something similiar for a suggestion.
I have attached a copy of removepkg from Slackware 13.1 that appears to
be a 1 clause BSD license ;) for reference.
As well I would also like to see something done about packages that
don't need to be upgraded because they are neither platform or arch
dependent but yet they are upgraded due to being listed as a dependent
of another port that needs to be upgraded. For example any package that
may be type shell script does not need updating due to a major lib
version bump of for say libpng.
Regards,
--
jhell,v
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
#!/bin/sh
# Slackware remove package script
#
# Sat Apr 25 21:18:53 UTC 2009 (12.34567890b)
# Converted to use new pkgbase() function to remove pathname and
# valid package extensions.
#
# Revision 12.34567890 Sun Apr 5 20:59:32 CDT 2009 volkerdi
# - Support packages with the extensions: .tgz, .tbz, .tlz, .txz
#
# Revision 1.9 Wed Oct 31 14:04:28 CDT 2007 volkerding
# - Fix problem removing packages with a large number of fields.
# Thanks to Niki Kovacs for noticing this, and to Piter Punk
# for the patch.
# - Use LC_ALL=C locale, which is much faster with sort.
# Thanks to Tsomi.
# - Don't try to remove any package that starts with '-'. This
# is not a proper package name (usually a typo), and results
# in the package database being broken. Thanks to Jef Oliver.
# - Patched cat_except() to allow the last Slackware package on
# a partition to be removed (using ROOT=, of course)
# Thanks to Selkfoster for the patch, and to everyone else who
# proposed solutions before. This issue really wasn't given
# the highest priority before, but I figured while I'm in here...
#
# Revision 1.8 Thu Nov 22 14:00:13 PST 2001 volkerding Rel $
# - Move $TMP underneath $ROOT
# - Understand the idea of a base package name, so that packages
# can be removed with any of these notations:
# removepkg foo-1.0-i386-1.tgz
# removepkg foo-1.0-i386-1
# removepkg foo.tgz
# removepkg foo
#
# Revision 1.7 2001/03/30 12:36:28 volkerding
# - Strip extra .tgz from input names.
#
# Revision 1.6 1999/03/25 18:26:41 volkerding
# - Use external $ROOT variable, like installpkg.
#
# Revision 1.5.1 1998/03/18 15:37:28 volkerding
# - Since removepkg is always run by root, the temp directory has been
# moved from /tmp to a private directory to avoid symlink attacks from
# malicious users.
#
# Revision 1.5 1997/06/26 12:09:53 franke
# - Fixed old bug in TRIGGER regex setting
# - -preserve/-copy options now preserve non-unique files
# and empty directories also
#
# Revision 1.4 1997/06/09 13:21:36 franke
# - Package file preserve (-preserve, -copy) added.
# - Don't execute rm -rf lines from doinst.sh, removing links explicit.
# - Warning on no longer existing files added.
# - Warning on files changed after package installation added.
# - Intermediate file preserve (-keep) added.
# - Check for required files/links now done on a combined list.
# - Write access to /var/log/{packages,scripts} no longer necessary for -warn.
#
# Revision 1.3 1997/06/08 13:03:05 franke
# Merged with revision 1.1.1.1
#
# Revision 1.2 1996/06/01 20:04:26 franke
# Delete empty directories formated manual pages added
#
# Revision 1.1.1.1 1995/12/18 21:20:42 volkerding
# Original Version from Slackware 3.1
#
# Revision 1.1 1995/06/05 22:49:11 volkerding
# Original Version from Slackware 3.0
#
# Copyright 1994, 1995, 1998 Patrick Volkerding,
Re: what next for the pkg_install rewrite
Le Jeu 19 aoû 10 à 23:24:10 +0200, jhell jh...@dataix.net écrivait : As well I would also like to see something done about packages that don't need to be upgraded because they are neither platform or arch dependent but yet they are upgraded due to being listed as a dependent of another port that needs to be upgraded. For example any package that may be type shell script does not need updating due to a major lib version bump of for say libpng. For this purpose, it's already possible to check if the port set NO_BUILD. Regards, -- Th. Thomas. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On 19/08/2010, jhell jh...@dataix.net wrote:
Adding to this I would like to see a central database created for
packages that have been removed like in Slackware Linux. They keep a
file in /var/log/preserved_packages with a flat text format with the
file name looking like:
${PORTNAME}-${PORTVERSION}${PORTREVISION}-`date +%Y%m%d%H%M%S`
Ah yes, you reminded me of this other thing: I would also suggest
getting rid of text files carrying rich information in ad-hoc formats
:)
I'm not saying XML should be the only choice, but it *is* well
supported - expat is even in base as libbsdxml.
While suggesting nebulous things I know will be hard to pass near a
lot of people: sqlite is *the* choice for any record-based file
databases today. The single most important thing I'll promote with it
is its transaction capabilities and ACID - these would get much use if
parallel operations (upgrades / installs) are to be supported. There
are a ton of other reasons too.
I started writing this a long time ago but abandoned it because of
strong opposition: http://wiki.freebsd.org/PortsUsingSQLite - maybe it
would help at this time.
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On Thu, Aug 19, 2010 at 6:10 PM, Ivan Voras ivo...@freebsd.org wrote:
On 19/08/2010, jhell jh...@dataix.net wrote:
Adding to this I would like to see a central database created for
packages that have been removed like in Slackware Linux. They keep a
file in /var/log/preserved_packages with a flat text format with the
file name looking like:
${PORTNAME}-${PORTVERSION}${PORTREVISION}-`date +%Y%m%d%H%M%S`
Ah yes, you reminded me of this other thing: I would also suggest
getting rid of text files carrying rich information in ad-hoc formats
:)
I'm not saying XML should be the only choice, but it *is* well
supported - expat is even in base as libbsdxml.
That might be acceptable.
While suggesting nebulous things I know will be hard to pass near a
lot of people: sqlite is *the* choice for any record-based file
databases today. The single most important thing I'll promote with it
is its transaction capabilities and ACID - these would get much use if
parallel operations (upgrades / installs) are to be supported. There
are a ton of other reasons too.
On the other hand, I have strong philosophical objections to core
port/package management utilities requiring large support structures
of other ports/packages.
Jim
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: what next for the pkg_install rewrite
On Thu, Aug 19, 2010 at 3:10 PM, Ivan Voras ivo...@freebsd.org wrote:
On 19/08/2010, jhell jh...@dataix.net wrote:
Adding to this I would like to see a central database created for
packages that have been removed like in Slackware Linux. They keep a
file in /var/log/preserved_packages with a flat text format with the
file name looking like:
${PORTNAME}-${PORTVERSION}${PORTREVISION}-`date +%Y%m%d%H%M%S`
Please no. We need another ad hoc text format like we need holes in our head.
Ah yes, you reminded me of this other thing: I would also suggest
getting rid of text files carrying rich information in ad-hoc formats
:)
I'm not saying XML should be the only choice, but it *is* well
supported - expat is even in base as libbsdxml.
While suggesting nebulous things I know will be hard to pass near a
lot of people: sqlite is *the* choice for any record-based file
databases today. The single most important thing I'll promote with it
is its transaction capabilities and ACID - these would get much use if
parallel operations (upgrades / installs) are to be supported. There
are a ton of other reasons too.
I started writing this a long time ago but abandoned it because of
strong opposition: http://wiki.freebsd.org/PortsUsingSQLite - maybe it
would help at this time.
There are a lot of ideas going around, but unless the current patches
get polished up and portmgr (points in flz's general direction)
actually accept ideas, ideas are nothing more than ideas and will
essentially be vaporware. That has been the chokepoint for patches
I've added to the PR database.
Also, if it breaks backwards compatibility used everywhere over the
map and is actually functionality that's used in ports and src
packages without equivalent functionality, it's a non-starter.
If any work needs to be done, we should be getting _rid_ of features
which aren't in use, or can be easily replaced with equivalent
methods. Stuff I've identified over the past year that can be worked
on are:
pkg_install
Locking:
- We need concurrent package installation.
Packing list:
@dirrmtry (needs to be replaced with scripting infrastructure called
via +INSTALL/+DEINSTALL). See @exec/@unexec (similar problem).
@exec/@unexec (needs to be replaced with scripting infrastructure
using +INSTALL/+DEINSTALL).
@group/@mode/@owner (needs to be replaced with scripting infrastructure).
@srcdir really shouldn't be in a plist (someone can script the call
easily with pkg_create).
At the end of the day, it would be really nice if there was one small
file with the package metadata, and the other file was checksums via
an mtree file. mtree files will solve a lot of the problems with the
ad hoc plist `format'.
@noinst should be checked to see whether or not it's widely used in
ports. It might be a good idea to implement, but I'm not confident in
that statement.
Dependencies:
- We should be using reverse dependencies, not forward dependencies.
There's less value in forward dependencies, because I can uninstall
things, and I don't have an accurate idea of what the current state is
on the system. Reverse dependencies actually tell us _what_ we depend
on, and should be expressed in terms of origins, not package versions.
Whenever the version for the origins change, the package should be
bumped. This would (IMO) reduce a lot of headaches with packages.
INDEX*:
- We should depend on INDEX* for package data, and it should be
distributed with the base system, not ports, and ports should build
off of this data.
*libpkg*!!:
- I emphasize this, because it's extremely important... David/Florent
are already doing work to make this a reality, BUT we shouldn't be
duplicating their work. It would be nice if the work being done by
both Florent and David was more transparent, could be tasked out to
various individuals, etc... but that's not how it is today.
Ports
bsd.*.mk:
- Needs to use pkg_add / pkg_create out of the box instead of
installing things directly into DESTDIR/LOCALBASE. This would
avoid a lot of issues with corrupted installs, etc.
- [Nice to have] should be trimmed wherever possible. There's a lot of
cruft that actually could be moved elsewhere to reduce the amount of
work in terms of parsing and evaluating statements for everyday ports
users.
In general:
- Ports need to be fixed so that BUILD_DEPENDS and RUN_DEPENDS don't
end up in the package dependency list. We suffer from this in areas,
Redhat suffers from this, etc, and it's not an easy task to do.
However, if done properly, this will 1) speed up package building, 2)
package installation, 3) reduce installed package count, etc.
Just for the sake of not repeating past discussions:
1. SQLite was killed before because of complexity and because it was
needs another package in base that isn't BSD licensed. That's why
everyone in the know has been pushing for BDB 1.8x (in base). People
suggested that to me back when I was trying to get off the ground in
GSoC
Re: what next for the pkg_install rewrite
On 08/19/2010 23:08, Garrett Cooper wrote:
On Aug 19, 2010, at 7:30 PM, jhell wrote:
On 08/19/2010 21:26, Garrett Cooper wrote:
On Thu, Aug 19, 2010 at 3:10 PM, Ivan Voras ivo...@freebsd.org
wrote:
On 19/08/2010, jhell jh...@dataix.net wrote:
Adding to this I would like to see a central database
created for packages that have been removed like in
Slackware Linux. They keep a file in
/var/log/preserved_packages with a flat text format with
the file name looking like:
${PORTNAME}-${PORTVERSION}${PORTREVISION}-`date
+%Y%m%d%H%M%S`
Please no. We need another ad hoc text format like we need holes
in our head.
You may have misunderstood or maybe not the intention behind that
file.
This is just simply a log file of the transactions that were
performed upon package deletion and nothing more but just a way for
the user to look back and say HEY! how did that get there!. or
where in the ``jhell'' did that file come from! that they can
simply grep the package removal logs to find out.
-- Shameless plug with my email name put in for humor.
:)
It is also really handy if you remove some packages that somehow
the depends had been messed up and later on your having a problem
with a missing lib, easy enough to grep the removal logs to find
out what package held that file. Especially useful if you only use
binary packages.
This is part of the request that someone was making for a feature
like Gentoo's world file on the forums:
http://forums.freebsd.org/showthread.php?t=16963 .
Personally it's one of the takeaways I like about Gentoo's portage
system because it's easy to track what I as a user installed
manually, and hence, it's easy to track what can be removed (instead
of using pkg_cutleaves) if we have a `emerge -C` (package dependency
[dist]clean equivalent). It also makes it easier for admins to mass
install packages on multiple machines using a smaller `distroot'
install binary base, so all I would have to do is:
1. Install prebuilt version of FreeBSD with sysinstall / ad hoc
installer method. 2. Say, pkg_add install all, force options 3. Do
whatever I need to do to configure the machine.
Done.
That would make system administration really easy and slick, and
would improve the setup process for corporations that build on a
static FreeBSD base for several releases and have varying packages
for several bits. I know if my group did it, things wouldn't be such
a mess..
By the way... /var/lib/portage/world is a simple text file composed
line by line like:
pkg-origin-a pkg-origin-b
i.e.
devel/gcc46 lang/python26 www/firefox36
etc. Simple and easy to understand, and easy to modify :).
There is a lot of information that can be logged, especially with
'-v', I personally do not think we or anyone for that matter should
pass up that opportunity to make sure the information is collected
rather than leaving it up to the user to redirect or script(1) the
output every time which they would still or should be able to do.
Another approach that I have not seen talked about here is a
proposed directory layout. I think before 'unless I missed it' that
someone jumps into this, some standards goals should be made and
made quite loudly as to attract as much public opinion and
suggestions as possible for what works, what does not what people
would like to see.
Something of this magnitude like changing packaging databases and
directory structures and all that involved needs a central place
and a clear, clean plan to be developed properly. I personally do
not see this list anymore as a proper place to discuss it.
packaging@ list request ? so this can all be centralized.
I agree that it's high time that a freebsd-packaging@ list be
created. sysinstall has its own list now -- we should have one for
the packaging software too :).
PS: I have been toying with the idea of the directory layout just
for spurring thoughts of others. http://bit.ly/aNLhNU but until
there is a central place for these things it does not mean much.
I think that you're adding unnecessary complexity to the overall
issue. It really doesn't make sense for me to install packages that
aren't available for my architecture for one (in particular today),
unless you were thinking of serving up this data on an NFS server,
but even then it doesn't make sense because almost all of these files
are hardcoded to exist at ${LOCALBASE} when built as ports, so
setting it to another location would be problematic. Other things
would need to be done before you could get to this stage.
Thinking of not only a bootp server here that serves up or can serve up
a mass amount of different machines it would make it possible for them
to have local-amd64, local-i386 served off to specific machines with
specific package databases. But then again that type of complexity is
probably not needed as a whole but is pretty keen.
Also I was building upon the idea of getting the
Re: what next for the pkg_install rewrite
On Aug 19, 2010, at 7:30 PM, jhell wrote:
On 08/19/2010 21:26, Garrett Cooper wrote:
On Thu, Aug 19, 2010 at 3:10 PM, Ivan Voras ivo...@freebsd.org wrote:
On 19/08/2010, jhell jh...@dataix.net wrote:
Adding to this I would like to see a central database created for
packages that have been removed like in Slackware Linux. They keep a
file in /var/log/preserved_packages with a flat text format with the
file name looking like:
${PORTNAME}-${PORTVERSION}${PORTREVISION}-`date +%Y%m%d%H%M%S`
Please no. We need another ad hoc text format like we need holes in our head.
You may have misunderstood or maybe not the intention behind that file.
This is just simply a log file of the transactions that were performed
upon package deletion and nothing more but just a way for the user to
look back and say HEY! how did that get there!. or where in the
``jhell'' did that file come from! that they can simply grep the
package removal logs to find out.
-- Shameless plug with my email name put in for humor.
:)
It is also really handy if you remove some packages that somehow the
depends had been messed up and later on your having a problem with a
missing lib, easy enough to grep the removal logs to find out what
package held that file. Especially useful if you only use binary packages.
This is part of the request that someone was making for a feature like Gentoo's
world file on the forums: http://forums.freebsd.org/showthread.php?t=16963 .
Personally it's one of the takeaways I like about Gentoo's portage system
because it's easy to track what I as a user installed manually, and hence, it's
easy to track what can be removed (instead of using pkg_cutleaves) if we have a
`emerge -C` (package dependency [dist]clean equivalent). It also makes it
easier for admins to mass install packages on multiple machines using a smaller
`distroot' install binary base, so all I would have to do is:
1. Install prebuilt version of FreeBSD with sysinstall / ad hoc installer
method.
2. Say, pkg_add install all, force options
3. Do whatever I need to do to configure the machine.
Done.
That would make system administration really easy and slick, and would improve
the setup process for corporations that build on a static FreeBSD base for
several releases and have varying packages for several bits. I know if my group
did it, things wouldn't be such a mess..
By the way... /var/lib/portage/world is a simple text file composed line by
line like:
pkg-origin-a
pkg-origin-b
i.e.
devel/gcc46
lang/python26
www/firefox36
etc. Simple and easy to understand, and easy to modify :).
There is a lot of information that can be logged, especially with '-v',
I personally do not think we or anyone for that matter should pass up
that opportunity to make sure the information is collected rather than
leaving it up to the user to redirect or script(1) the output every time
which they would still or should be able to do.
Another approach that I have not seen talked about here is a proposed
directory layout. I think before 'unless I missed it' that someone jumps
into this, some standards goals should be made and made quite loudly
as to attract as much public opinion and suggestions as possible for
what works, what does not what people would like to see.
Something of this magnitude like changing packaging databases and
directory structures and all that involved needs a central place and a
clear, clean plan to be developed properly. I personally do not see this
list anymore as a proper place to discuss it. packaging@ list request ?
so this can all be centralized.
I agree that it's high time that a freebsd-packaging@ list be created.
sysinstall has its own list now -- we should have one for the packaging
software too :).
PS: I have been toying with the idea of the directory layout just for
spurring thoughts of others. http://bit.ly/aNLhNU but until there is a
central place for these things it does not mean much.
I think that you're adding unnecessary complexity to the overall issue. It
really doesn't make sense for me to install packages that aren't available for
my architecture for one (in particular today), unless you were thinking of
serving up this data on an NFS server, but even then it doesn't make sense
because almost all of these files are hardcoded to exist at ${LOCALBASE} when
built as ports, so setting it to another location would be problematic. Other
things would need to be done before you could get to this stage.
Also, many of the ports installed are prefixed with the package name, which is
different for multiple ports. Example:
$ ls /var/db/pkg/python*
python-2.6,2/ python26-2.6.5_1/ python31-3.1.2_1/
Having a concept of multiple versions in ports would require a major overhaul
to get things to work in a Gentoo like method, and I'm not sure how many people
would be particularly keen in doing this (especially when there are name
