Re: ports and PBIs
On Sunday, April 11, 2010, Tim Kientzle kient...@freebsd.org wrote: Garrett Cooper wrote: If I'm understanding you correctly you're saying it's an issue when I do: pkg_add A B C # 1 year passes pkg_add D # D depends on A, B, C, of different revisions. pkg_add barfs because it can't find the applications, etc. This is something that's been hashed over a number of times (a few of which I've participated in in #bsdports). There needs to be a simple update command which will handle the action of upgrading packages, because there isn't a proper command that will do so today. I'm not convinced that the simple update command you mention is actually feasible, much less desirable. (If I want to try out the new Firefox, why does that imply that my year-old Gimp has to be upgraded?) As for feasibility, here's the easy problem: A2.7 requires B3.6 ... one year passes ... A4.8 now requires B7.2 But A4.8 is incompatible with B3.6 and A2.7 is incompatible with B7.2. So neither A nor B can be updated separately without breaking the system. Here's the hard problem: A2.7 requires B3.6 ... one year passes ... I want to install C1.0 which requires B7.2 but there hasn't been a new release of A that works with B7.2. So I now simply cannot have both C1.0 and A2.7 installed at the same time because they require different versions of B. PBI avoids both of these problems. It may be unsuitable for embedded systems[1], but I see no reason we should not extend the existing ports/packages system with additional tools that target certain use cases, and PBI seems a good fit for the desktop case. Tim Genuine (possibly stupid) question -in PBI land, what happens if package B is, say, CUPS? Does one need versioned rc.d scripts to start one or the other? Which one gets to claim port 631? -James Butler [1] Actually, PBI might work just fine even for embedded if we address the disk bloat issue. One approach would be to make /Package/Bar/libfoo-2.8.7.so a symlink or hardlink to /Package/Shared/libfoo-2.8.7.so-MD5-hash This gives easy sharing of identical files. It's even easy to handle at install time: * Installer writes libfoo-2.8.7.so to /Package/Shared/libfoo-2.8.7.so-temp-PID of installer * Installer computes hash of file as it's written * Installer renames file (delete if rename fails with EEXIST) * Installer writes symlink or hardlink into /Package/Bar ___ freebsd-curr...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On 4/10/2010 3:18 PM, k...@pcbsd.org wrote snip However for my more hard-core friends, nothing stopping you from running your own ports down the road, more power to ya! For doing something like embedded work or a server this makes total sense and I think it is a huge positive for FreeBSD, no reason to trash that or break it in any way. For the other 99.9% of society who want something that just works for day-to-day computing, something like PBI is very attractive. It would be great to have an OS that offers best of both worlds. -- Kris Moore There are only two possibilities with any package system. Either give the user self packaged binaries containing all shared libraries or make them update everything. Both have positives and negatives. We've been working on a new package system in MidnightBSD for some time. When we weighed this issue, it was decided that letting users have old binaries sitting around was a bad idea. It encourages a user to sit on a package for a year and not install security updates. The larger package size also deters users from downloading updates in parts of the world which have slow Internet connections. Remember the GDI+ update to windows awhile back? There were many applications that had to be updated and Microsoft had to release a scanner to search the drive for uses. There side isn't always rosy. Obviously, there are also advantages to the larger PBI packages for users. PC-BSD is certainly easy to use. At the end of the day, I think creating packages more frequently during releases and pushing updates like many linux distros do makes more sense in terms of security. FreeBSD has ten times the number of ports to build than we do so obviously it's a problem to build packages that frequently. I don't want to butt in any more on this because it's not my place, but I just felt it was important to bring another perspective. Lucas ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On 4/13/10 12:09 AM, Lucas Holt wrote: On 4/10/2010 3:18 PM, k...@pcbsd.org wrote snip However for my more hard-core friends, nothing stopping you from running your own ports down the road, more power to ya! For doing something like embedded work or a server this makes total sense and I think it is a huge positive for FreeBSD, no reason to trash that or break it in any way. For the other 99.9% of society who want something that just works for day-to-day computing, something like PBI is very attractive. It would be great to have an OS that offers best of both worlds. -- Kris Moore There are only two possibilities with any package system. Either give the user self packaged binaries containing all shared libraries or make them update everything. Both have positives and negatives. We've been working on a new package system in MidnightBSD for some time. When we weighed this issue, it was decided that letting users have old binaries sitting around was a bad idea. It encourages a user to sit on a package for a year and not install security updates. The larger package size also deters users from downloading updates in parts of the world which have slow Internet connections. Remember the GDI+ update to windows awhile back? There were many applications that had to be updated and Microsoft had to release a scanner to search the drive for uses. There side isn't always rosy. Obviously, there are also advantages to the larger PBI packages for users. PC-BSD is certainly easy to use. At the end of the day, I think creating packages more frequently during releases and pushing updates like many linux distros do makes more sense in terms of security. FreeBSD has ten times the number of ports to build than we do so obviously it's a problem to build packages that frequently. I don't want to butt in any more on this because it's not my place, but I just felt it was important to bring another perspective. It may be thaat part of the answer is to do both. For me I want to have PBIs for the actual tools I use on the machine.. things like wine, openoffice, gimp, etc. I don't care if these are 'bleeding edge'. I just want them to work, and to keep working no matter what I do in my development environment. On the other had for stuff I'm working on, I want ot be able to get the newest libraries etc and keep them up to date. This means I run the dependency problem but I'm willing to upgrade everything and if it breaks occasionally, I'll fix it. regardless of whether my development environment is current;y broke or not, the tools I actually use on the machine will not be affected. So for me I see a reason tehat we should use BOTH schemes. Lucas ___ freebsd-curr...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On 13/04/10 19:09, Lucas Holt wrote: On 4/10/2010 3:18 PM, k...@pcbsd.org wrote snip However for my more hard-core friends, nothing stopping you from running your own ports down the road, more power to ya! For doing something like embedded work or a server this makes total sense and I think it is a huge positive for FreeBSD, no reason to trash that or break it in any way. For the other 99.9% of society who want something that just works for day-to-day computing, something like PBI is very attractive. It would be great to have an OS that offers best of both worlds. -- Kris Moore There are only two possibilities with any package system. Either give the user self packaged binaries containing all shared libraries or make them update everything. Both have positives and negatives. We've been Only two possibilities? I remember reading about a third in a paper proposing an enhancement to pkgsrc, but now I can't find the paper :-( Basically this involves distributing and downloading separate packages as with pkg_add, but installing each one in an isolated directory so multiple conflicting versions can be present side-by-side. The package tools then arrange for each package to have symlinks to all the packages it depends on, in a private libdepends/ folder or something. So if I install package A1.1 which depends on B3.2, then I want to install C5.3 which depends on B4.0, I have a choice: to upgrade A to a version depending on B4.0 if one exists, or to install C5.3 with B4.0 alongside A1.1 and B3.2. I'll try to find the paper I'm recalling after work, until then I'll just have to wave my hands in the air. -James Butler working on a new package system in MidnightBSD for some time. When we weighed this issue, it was decided that letting users have old binaries sitting around was a bad idea. It encourages a user to sit on a package for a year and not install security updates. The larger package size also deters users from downloading updates in parts of the world which have slow Internet connections. Remember the GDI+ update to windows awhile back? There were many applications that had to be updated and Microsoft had to release a scanner to search the drive for uses. There side isn't always rosy. Obviously, there are also advantages to the larger PBI packages for users. PC-BSD is certainly easy to use. At the end of the day, I think creating packages more frequently during releases and pushing updates like many linux distros do makes more sense in terms of security. FreeBSD has ten times the number of ports to build than we do so obviously it's a problem to build packages that frequently. I don't want to butt in any more on this because it's not my place, but I just felt it was important to bring another perspective. Lucas ___ freebsd-curr...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On Sun, Apr 11, 2010 at 03:44:37PM -0700, Julian Elischer wrote: On 4/11/10 12:20 PM, Kostik Belousov wrote: On Sun, Apr 11, 2010 at 12:13:12PM -0700, Julian Elischer wrote: On 4/11/10 11:44 AM, Kostik Belousov wrote: On Sun, Apr 11, 2010 at 11:23:33AM -0700, Julian Elischer wrote: On 4/11/10 3:27 AM, Kostik Belousov wrote: I already pointed in the other reply in this thread, $ORIGIN dynamic token should solve the issue. See http://docs.sun.com/app/docs/doc/817-1984/chapter3-13312?l=ena=view yes, teh question I have since I am not alinker expert is do we support it? the link you give is for Solaris I think.. It is in three for HEAD, RELENG_8 and RELENG_7. thank you. This will I think as you suggest, make a significant difference. the question I have is is it re-evaluated for each library? You interpreted the question correctly. I am not sure what exactly you are asking there. $ORIGIN is substituted for each object invividually, taking the object path as a substitution target. That is, if main executable A references dso $ORIGIN/X/libA.so, then libA.so is looked up in the subdirectory X of directory containing A. If libA.so references $ORIGIN/Y/libB.so, then libB.so is looked up in X/Y subdirectory of directory containing A. If there is an LDPATH set then if the library A is to be found at $SOMEWHERE-ELSE which is in the LDPATH, rather than in $ORIGIN/X, will it still be found? LD_LIBRARY_PATH ? I do not think this will work, because $ORIGIN substitution (mostly) results in the absolute pathname. It is complicated by the fact that you might do things like ../$ORIGIN/libA.so, but this is plain silly. if the answer to the above is yes, then If it is then found in $SOMEWHERE-ELSE/X somewhere, will it then look for libB.so in $ORIGIN(A) or in $SOMEWHERE-ELSE ? Regardeless of the answer for the first question, $ORIGIN is evaluated. If the library is actually somewhere else (via symlink) is $origin reevaluated to the actual destination? (that would be cool). No. So, to recap: what we were thinking is something along the lines of the following: an example with 2 PBI apps created at the same time (part of the same set) application 1 libraryA - - (originally) - - -library B | / | |link / |link | /---(y)---/ | v / v common area dd-mm-yy library A --(x)library B ^^ |link|link || || application 1 libraryA - - (originally) - - - -library B library A and B in app 2 are deleted and libraries A and B in common area can be updated for security reasons by a special kind of PBI or package should it be required. It sounds to me that link 'y' is followed, i.e. the linker continues to think it is working in $ORIGIN(A). either way this is sounding very doable.. Kris is thinking about a single sysutils/pbimanager port and a /mk diff that would allow make pbi (once the port was installed). I think it actually looks quite feasible. Is there someone out there in ports-land who really inderstands the ports mk framework who could help us (because we'll need a local guide to make sure we don't dig inn any local burial grounds) and who can help with testing etc? Similarly if we need to do anything funny with regards to hashing parts of .so files, or deciding how to version things, is there an elf specialist in the house who can help? Kris said can do the pbi tools part if he has help with these two areas Julian pgpTFNSezrHcc.pgp Description: PGP signature
Re: ports and PBIs
On 4/10/10 10:06 PM, Garrett Cooper wrote: It's more than just diskspace though. Consider the fact that now you're going to lose a lot of the memory sharing between shared libs and what-not, and now you'd have to be running N number of daemons . Take PCBSD for instance -- do they really revision packages with unshared dependencies, or are all of the dependencies updated at once? This becomes a big issue as you can't have dissimilar applications like dbus, gamin, openssh, etc running on the same system at one time. How does the PBI layout plan to deal with this kind of conflict -- apart from jails, which would greatly increase the required footprint...? It's a pitty that you didn't read the original post where it was stated that doing this would depend on a scheme that is under discussion for common components to be shared WHEN POSSIBLE. If you can do this with package code, Maybe you will supply the packages.. Not quite sure what you meant here. I meant. get involved and do some of the work if you can see such an easy answer. why? As people have said before.. embedded folks usually want to compile everyrthing for themselves anyhow. Not necessarily. You have folks in embedded rolling their own stuff, sure, but then you have groups like Montavista (now owned by Cavium Networks) repackaging Linux for customers, providing a nominal fee for the packages, support, and the tools, and there are large companies (like Cisco) buying into this. It's not to say that people are going to not roll their own solution, but many [intelligent] folks are going towards an externally supported model instead of rolling their own stuff. Thus, whatever the community decides is sane is what gets adopted (unless the developers or management for the group are really foolhardy / ignorant of what exists in the outside world, they're steeped in existing methods that can't be easily transitioned to the new model, or they have expendable resources to toss towards a custom solution for specific needs). If you guys think PBIs are so great... tell ya what -- make me and other folks believers: You know young fellow, your attitude is kind of annoying for a topic that is just up for discussion. 1. Produce a port with the magic PBI producing tool. I hope to be able to do this soon. 2. Produce directions on how to use said tool. the goal is: cd /usr/ports/misc/cowsay (or whereever cowsay is) make pbi 3. Make sure said tool and install method doesn't conflict with what exists in base. PBIs already don't conflict. Hav eyou ever tried PC-BSD? 4. Capture statistics of how many people download this stuff and use well we would start with the number of people using PC-BSD because if we did this they would use our stuff. 5. Come back when you have data proving how many people care for your solution so portmgr and core can make an informed decision on whether or not it should be a part of base. that's not how it's ever worked and I doubt it's going to start now. Oh, and think about this too: whoever produces the tool, eats the support costs. The project shouldn't eat the support costs until it's a part of base, if that ever happens. Definitely take this point into consideration because good support is not only `my package/port/PBI is broken .. help me!' -- it's also having QA engineers on hand and staffed to validate that the packages or PBIs are valid and functional -- in the very least from a DoA / smoke test perspective. I realize that this is lacking in packages / ports today, but it's something that many folks volunteering in the project (cross-functional in bugs area and also in ports) have wanted for a long time. Sorry, but you've really pissed me off and as most people will tell you. that's not easy to do. This all has the ring of a desperate person looking for excuses to complain about something. Every thing you have mentioned occurred to those of us having the original discussion in about the oh, say FIRST 10 SECONDS of the conversation. Might I suggest that when you have been in the project another decade or so you might learn some manners and stop trying to teach you grandmother to suck eggs. If you are trying to tell me about project dynamics or how things work of need to work I put it to you that I've been doing this when you were about minus 10 years old. I do Not need a lecture from a wet behind the ears puppy about how I should handle a discussion with interested parties on possibly improving FreeBSD's user experience. When you were born I was decoding network traces. When you were giving you mother heart attacks by eating the crayons I was writing disk and network drivers for BSD and long haul protocols. When you were learning to read I was playing with the MACH VM system and kerle build process. When you were learning to do multiplication of small numbers I trying to forget the Windows NT internals classes I had been sent to. Do you think we are so stupid that we
Re: ports and PBIs
On Sat, Apr 10, 2010 at 03:45:20PM -0700, Tim Kientzle wrote: Julian Elischer wrote: On 4/10/10 12:07 PM, Tim Kientzle wrote: [1] Actually, PBI might work just fine even for embedded if we address the disk bloat issue. One approach would be to make /Package/Bar/libfoo-2.8.7.so a symlink or hardlink to /Package/Shared/libfoo-2.8.7.so-MD5-hash This gives easy sharing of identical files. yeah that's more or less what we were thinking.. hardlinks allow you to garbage collect when the last pbi that needs something is replaced/removed. The point of /Package/Shared in this design is basically that it provides a list of all of the files that can be shared, so you avoid doing a full disk search to identify other places that might have this file. You could accomplish the same goal by building and storing a database of sharable files somewhere, of course. (Curiously, no one has mentioned filesystem-level deduping yet as the big hammer solution... ;-) The LD_LIBRARY_PATH issue is the most interesting problem here. I don't immediately see a solution that doesn't include teaching ld-elf.so.1 about some form of per-application library path. I already pointed in the other reply in this thread, $ORIGIN dynamic token should solve the issue. See http://docs.sun.com/app/docs/doc/817-1984/chapter3-13312?l=ena=view pgp8kblV37kSh.pgp Description: PGP signature
Re: ports and PBIs
On Sunday, April 11, 2010, Tim Kientzle kient...@freebsd.org wrote: Garrett Cooper wrote: If I'm understanding you correctly you're saying it's an issue when I do: pkg_add A B C # 1 year passes pkg_add D # D depends on A, B, C, of different revisions. pkg_add barfs because it can't find the applications, etc. This is something that's been hashed over a number of times (a few of which I've participated in in #bsdports). There needs to be a simple update command which will handle the action of upgrading packages, because there isn't a proper command that will do so today. I'm not convinced that the simple update command you mention is actually feasible, much less desirable. (If I want to try out the new Firefox, why does that imply that my year-old Gimp has to be upgraded?) As for feasibility, here's the easy problem: A2.7 requires B3.6 ... one year passes ... A4.8 now requires B7.2 But A4.8 is incompatible with B3.6 and A2.7 is incompatible with B7.2. So neither A nor B can be updated separately without breaking the system. Here's the hard problem: A2.7 requires B3.6 ... one year passes ... I want to install C1.0 which requires B7.2 but there hasn't been a new release of A that works with B7.2. So I now simply cannot have both C1.0 and A2.7 installed at the same time because they require different versions of B. PBI avoids both of these problems. It may be unsuitable for embedded systems[1], but I see no reason we should not extend the existing ports/packages system with additional tools that target certain use cases, and PBI seems a good fit for the desktop case. Tim Genuine (possibly stupid) question -in PBI land, what happens if package B is, say, CUPS? Does one need versioned rc.d scripts to start one or the other? Which one gets to claim port 631? -James Butler [1] Actually, PBI might work just fine even for embedded if we address the disk bloat issue. One approach would be to make /Package/Bar/libfoo-2.8.7.so a symlink or hardlink to /Package/Shared/libfoo-2.8.7.so-MD5-hash This gives easy sharing of identical files. It's even easy to handle at install time: * Installer writes libfoo-2.8.7.so to /Package/Shared/libfoo-2.8.7.so-temp-PID of installer * Installer computes hash of file as it's written * Installer renames file (delete if rename fails with EEXIST) * Installer writes symlink or hardlink into /Package/Bar ___ freebsd-curr...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On 4/11/10 3:27 AM, Kostik Belousov wrote: I already pointed in the other reply in this thread, $ORIGIN dynamic token should solve the issue. See http://docs.sun.com/app/docs/doc/817-1984/chapter3-13312?l=ena=view yes, teh question I have since I am not alinker expert is do we support it? the link you give is for Solaris I think.. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On Sun, Apr 11, 2010 at 11:23:33AM -0700, Julian Elischer wrote: On 4/11/10 3:27 AM, Kostik Belousov wrote: I already pointed in the other reply in this thread, $ORIGIN dynamic token should solve the issue. See http://docs.sun.com/app/docs/doc/817-1984/chapter3-13312?l=ena=view yes, teh question I have since I am not alinker expert is do we support it? the link you give is for Solaris I think.. It is in three for HEAD, RELENG_8 and RELENG_7. pgpam1Yk4Iu1q.pgp Description: PGP signature
Re: ports and PBIs
On 4/11/10 11:44 AM, Kostik Belousov wrote: On Sun, Apr 11, 2010 at 11:23:33AM -0700, Julian Elischer wrote: On 4/11/10 3:27 AM, Kostik Belousov wrote: I already pointed in the other reply in this thread, $ORIGIN dynamic token should solve the issue. See http://docs.sun.com/app/docs/doc/817-1984/chapter3-13312?l=ena=view yes, teh question I have since I am not alinker expert is do we support it? the link you give is for Solaris I think.. It is in three for HEAD, RELENG_8 and RELENG_7. thank you. This will I think as you suggest, make a significant difference. the question I have is is it re-evaluated for each library? So, to recap: what we were thinking is something along the lines of the following: an example with 2 PBI apps created at the same time (part of the same set) application 1 libraryA - - (originally) - - -library B | / | |link / |link | /---(y)---/ | v / v common area dd-mm-yy library A --(x)library B ^^ |link|link || || application 1 libraryA - - (originally) - - - -library B library A and B in app 2 are deleted the idea that all the PBIs developed as part of a release set (labeled as set dd-mm-yy in this example, would have identical libraries in them and would thus be candidates for library consolidation. The question is and I guess it's not really that important, whether satisfaying a dependency in library A due to application 1 will use path (x) or path (y). certainly we would need to label the versions of the libraries in the common area with a hash or something, or maybe some other unique label. (port sequence number plus build args?) so that we don't use a copy of the library that is not really suitable for that app. A really top class effort would be ab;e to know hte set of build options on a library that would make the outcome acceptable. but I doubt that we want to go that far. if a person takes PBIS from set 01-01-2011 hey will tend to find common libraries. butit for some reason they need to take something from set 01-01-2009 (i.e. an old PBI, for some compatibility reason) it is guaranteed to work, despite the fact that there may well be collisions between library versions, because it will not be linked with those in the common area that do not match and will instead be linked with its own copies. Julian ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On Sun, Apr 11, 2010 at 12:13:12PM -0700, Julian Elischer wrote: On 4/11/10 11:44 AM, Kostik Belousov wrote: On Sun, Apr 11, 2010 at 11:23:33AM -0700, Julian Elischer wrote: On 4/11/10 3:27 AM, Kostik Belousov wrote: I already pointed in the other reply in this thread, $ORIGIN dynamic token should solve the issue. See http://docs.sun.com/app/docs/doc/817-1984/chapter3-13312?l=ena=view yes, teh question I have since I am not alinker expert is do we support it? the link you give is for Solaris I think.. It is in three for HEAD, RELENG_8 and RELENG_7. thank you. This will I think as you suggest, make a significant difference. the question I have is is it re-evaluated for each library? I am not sure what exactly you are asking there. $ORIGIN is substituted for each object invividually, taking the object path as a substitution target. That is, if main executable A references dso $ORIGIN/X/libA.so, then libA.so is looked up in the subdirectory X of directory containing A. If libA.so references $ORIGIN/Y/libB.so, then libB.so is looked up in X/Y subdirectory of directory containing A. So, to recap: what we were thinking is something along the lines of the following: an example with 2 PBI apps created at the same time (part of the same set) application 1 libraryA - - (originally) - - -library B | / | |link / |link | /---(y)---/ | v / v common area dd-mm-yy library A --(x)library B ^^ |link|link || || application 1 libraryA - - (originally) - - - -library B library A and B in app 2 are deleted the idea that all the PBIs developed as part of a release set (labeled as set dd-mm-yy in this example, would have identical libraries in them and would thus be candidates for library consolidation. The question is and I guess it's not really that important, whether satisfaying a dependency in library A due to application 1 will use path (x) or path (y). certainly we would need to label the versions of the libraries in the common area with a hash or something, or maybe some other unique label. (port sequence number plus build args?) so that we don't use a copy of the library that is not really suitable for that app. A really top class effort would be ab;e to know hte set of build options on a library that would make the outcome acceptable. but I doubt that we want to go that far. if a person takes PBIS from set 01-01-2011 hey will tend to find common libraries. butit for some reason they need to take something from set 01-01-2009 (i.e. an old PBI, for some compatibility reason) it is guaranteed to work, despite the fact that there may well be collisions between library versions, because it will not be linked with those in the common area that do not match and will instead be linked with its own copies. Julian pgpch2cIiNqfy.pgp Description: PGP signature
Re: ports and PBIs
On 4/11/10 12:20 PM, Kostik Belousov wrote: On Sun, Apr 11, 2010 at 12:13:12PM -0700, Julian Elischer wrote: On 4/11/10 11:44 AM, Kostik Belousov wrote: On Sun, Apr 11, 2010 at 11:23:33AM -0700, Julian Elischer wrote: On 4/11/10 3:27 AM, Kostik Belousov wrote: I already pointed in the other reply in this thread, $ORIGIN dynamic token should solve the issue. See http://docs.sun.com/app/docs/doc/817-1984/chapter3-13312?l=ena=view yes, teh question I have since I am not alinker expert is do we support it? the link you give is for Solaris I think.. It is in three for HEAD, RELENG_8 and RELENG_7. thank you. This will I think as you suggest, make a significant difference. the question I have is is it re-evaluated for each library? You interpreted the question correctly. I am not sure what exactly you are asking there. $ORIGIN is substituted for each object invividually, taking the object path as a substitution target. That is, if main executable A references dso $ORIGIN/X/libA.so, then libA.so is looked up in the subdirectory X of directory containing A. If libA.so references $ORIGIN/Y/libB.so, then libB.so is looked up in X/Y subdirectory of directory containing A. If there is an LDPATH set then if the library A is to be found at $SOMEWHERE-ELSE which is in the LDPATH, rather than in $ORIGIN/X, will it still be found? if the answer to the above is yes, then If it is then found in $SOMEWHERE-ELSE/X somewhere, will it then look for libB.so in $ORIGIN(A) or in $SOMEWHERE-ELSE ? If the library is actually somewhere else (via symlink) is $origin reevaluated to the actual destination? (that would be cool). So, to recap: what we were thinking is something along the lines of the following: an example with 2 PBI apps created at the same time (part of the same set) application 1 libraryA - - (originally) - - -library B | / | |link / |link | /---(y)---/ | v / v common area dd-mm-yy library A --(x)library B ^^ |link|link || || application 1 libraryA - - (originally) - - - -library B library A and B in app 2 are deleted and libraries A and B in common area can be updated for security reasons by a special kind of PBI or package should it be required. It sounds to me that link 'y' is followed, i.e. the linker continues to think it is working in $ORIGIN(A). either way this is sounding very doable.. Kris is thinking about a single sysutils/pbimanager port and a /mk diff that would allow make pbi (once the port was installed). I think it actually looks quite feasible. Is there someone out there in ports-land who really inderstands the ports mk framework who could help us (because we'll need a local guide to make sure we don't dig inn any local burial grounds) and who can help with testing etc? Similarly if we need to do anything funny with regards to hashing parts of .so files, or deciding how to version things, is there an elf specialist in the house who can help? Kris said can do the pbi tools part if he has help with these two areas Julian ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On Fri, Apr 9, 2010 at 8:31 PM, Julian Elischer jul...@elischer.org wrote: sorry for the cross-post.. Last night at the Bay Area FreeBSD Users Group meeting we had a discussion about ports, and what is good about them and what is bad about them. This has been a topic of discussion quite a bit recently and we were looking for a solution that would allow us to keep the good parts of the current ports system but would allow us to give a better user experience for non guru users. The scheme we came up with involves a merging of the ports tree and the PBI system, developed for PC-BSD. Basically, the addition of a makepbi keyword in the .mk files to allow the automatic generation of PBIs for 'simple' ports such as 'cowsay' (the canonical simple app). More complicated apps would need manual work in Makefile or in a separate pbi-recipe file, but once the support was done we could proceed one port at a time. Not all ports make sense in a PBI format. (e.g. libraries etc. may not) I for one support this Idea, and at a BoF FreeBSD Desktop session at BSDCan 2008 one of my suggestions was to have FreeBSD bless PBI's I think this is good For PC-BSD. and in return it is GREAT for FreeBSD, as it will widen the user base and hopefully attract a few more good developers. keep this discussion going, because there isn't mush of a downside so far as I can see. Sam Fourman Jr. One issue that was raised is the increase of storage overhead when using PBI packages as they include a copy of all required libraries and resources, which means that one would very quickly get duplicate copies of things. Our suggestions include the ability of the PBI management software to resolve and (using hard links) eliminate duplicate items. This is not as easy as it sounds but can be achieved using a special variant of 'objcopy' (at least that is our theory). The aim is to make all apps installed on a system much more resilient to dependency problems. In addition there was discussion on how builds need to be doable as non-root uids sometimes, and that users on a system should be able to install packages (PBIs) as thie selves to get local versions of apps for themselves. Alfred Perlstein , Matt at ix systems Kris (Mr PBI), some others and I, felt that these ideas seemed to make some sense and so I put them here for comment. Julian ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On Fri, Apr 9, 2010 at 10:11 PM, Adam Vande More amvandem...@gmail.com wrote: On Fri, Apr 9, 2010 at 8:31 PM, Julian Elischer jul...@elischer.org wrote: Alfred Perlstein , Matt at ix systems Kris (Mr PBI), some others and I, felt that these ideas seemed to make some sense and so I put them here for comment. FWIW, when I see these discussions I'm always left wondering what's the bad part? I do think there are problems, but there doesn't seem to be a clear defined set of what is wrong. IMO, there should be a defined set of goals to judge possible implementations against. Let me start by saying FreeBSD ports is by far the best system I have used to date. but as good as it is, there is room for improvement. Being a FreeBSD user now for many years, one thing I think would be nice is: being able to have easier access to development ports( Masked ports kinda like Gentoo). right now is a GREAT example, currently there are new Gnome ,KDE and Xorg. these are all MAJOR ports,dependencies run deeper and deeper with every release. there can never be enough testing...but they all exist in random subversion servers around the web... I would very much like to help test these Major ports, but installing them is a pain. there should be some sort of overlay system in place, so I can just build the development ports after agreeing to a few well placed warnings of course. and Well if I hose my system all to hell.. well then I could just click on a bunch of PBI's and I am back in business... better still, make the development ports a PBI, I am just thinking out loud here,but that may work, toughts? one could say I could use merge scripts like marcusmerge for example, or use Virtualbox... but for large ports like Xorg and gnome or KDE, virtualbox doesn't cut it yet... thinks like Nvidia Video cards, multiple monitors, USB devices, and whatnot do not work on virtual box.. PBI's for development ports, with all the dependencies, wrapped in one package. solution? well let all the developers develop working ports in progress in one place, give users like me a way to track these changes and install and test them... I think FreeBSD becomes a better place for it. Sam Fourman Jr. Fourman Networks ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On Sat, Apr 10, 2010 at 2:20 AM, Garrett Cooper yanef...@gmail.com wrote: On Fri, Apr 9, 2010 at 11:28 PM, Sam Fourman Jr. sfour...@gmail.com wrote: On Fri, Apr 9, 2010 at 10:11 PM, Adam Vande More amvandem...@gmail.com wrote: On Fri, Apr 9, 2010 at 8:31 PM, Julian Elischer jul...@elischer.org wrote: Alfred Perlstein , Matt at ix systems Kris (Mr PBI), some others and I, felt that these ideas seemed to make some sense and so I put them here for comment. FWIW, when I see these discussions I'm always left wondering what's the bad part? I do think there are problems, but there doesn't seem to be a clear defined set of what is wrong. IMO, there should be a defined set of goals to judge possible implementations against. Let me start by saying FreeBSD ports is by far the best system I have used to date. but as good as it is, there is room for improvement. Being a FreeBSD user now for many years, one thing I think would be nice is: being able to have easier access to development ports( Masked ports kinda like Gentoo). Masking ports and packages in general introduces all sorts of fun new complexity for end users as well as maintainers. The last time I used Gentoo (which was only a matter of months ago), a lot portage packages were still masked even though they've been stable for months, years, etc. This is very annoying for me as an end-user because bug blah could be fixed in a later release but in order to unmask the pieces for version blah, I had to unmask 10~15 other `unstable packages', which greatly increased the chance of instability on my system (this was particularly the case back several years ago, but Gentoo has become more conservative over the years, and appears to be approaching some level of equilibrium with Fedora, Ubuntu, etc in terms of releases and package versioning). I wasn't suggesting that the current way Gentoo did Masking was the correct way, in fact you have valid points that I agree with and I used Gentoo last Week :) What I like is that, most of the portage development in done in tree, maybe the real solution is, to just have a development and release ports tree? right now is a GREAT example, currently there are new Gnome ,KDE and Xorg. these are all MAJOR ports,dependencies run deeper and deeper with every release. there can never be enough testing...but they all exist in random subversion servers around the web... ports isn't going to solve this. Post the Xorg modularization (which needed to occur anyhow because Xorg and Xfree86 before that was were monolithic beasts), I personally don't see that change in the amount of flux on a quarterly cycle, and the number of packages I install today isn't that much greater than back 6 years ago when I started using FreeBSD. So, while there might be some claim here to note, I think it's mostly exaggerated. Again, I agree with you, I just want a easier way to test these large ports. I would very much like to help test these Major ports, but installing them is a pain. there should be some sort of overlay system in place, so I can just build the development ports after agreeing to a few well placed warnings of course. and Well if I hose my system all to hell.. well then I could just click on a bunch of PBI's and I am back in business... Ok, apart from the interface (click a PBI, and magically you have packages installed)... how is this really different from binary packages? Have you tried installing binary packages lately via pkg_add? If not, I'd give it a shot instead of installing from ports. pkg_add does work, I have done it several times, upon learning about PBI's a few years back I wondered to myself, why not just use packages,and make some sort of GUI to add a icon to the whole ordeal. but now I get the Idea of dependencies,it pleges evey Open source OS, even ubuntu breaks every now and again. better still, make the development ports a PBI, I am just thinking out loud here,but that may work, toughts? one could say I could use merge scripts like marcusmerge for example, or use Virtualbox... but for large ports like Xorg and gnome or KDE, virtualbox doesn't cut it yet... thinks like Nvidia Video cards, multiple monitors, USB devices, and whatnot do not work on virtual box.. PBI's for development ports, with all the dependencies, wrapped in one package. Ok, well here's the thing. Instead of having N shared dependencies and libraries in /usr/local/lib, you'd have N**2 shared dependencies and libraries in each and every package. Now, let's look at size difference. Here's just one sample: $ ls -l irssi-0.8.14_1.tbz ~/Downloads/Irssi0.8.14_1-PV0.pbi -rw-r--r-- 1 gcooper gcooper 6856203 Apr 10 00:05 /usr/home/gcooper/Downloads/Irssi0.8.14_1-PV0.pbi -rw-r--r-- 1 root wheel 517442 Apr 10 00:07 irssi-0.8.14_1.tbz The .tbz file is a file created with pkg_create -b, and the other file is the PBI I pulled off of
Re: ports and PBIs
On 4/10/10 12:20 AM, Garrett Cooper wrote: On Fri, Apr 9, 2010 at 11:28 PM, Sam Fourman Jr.sfour...@gmail.com wrote: On Fri, Apr 9, 2010 at 10:11 PM, Adam Vande Moreamvandem...@gmail.com wrote: On Fri, Apr 9, 2010 at 8:31 PM, Julian Elischerjul...@elischer.org wrote: Alfred Perlstein , Matt at ix systems Kris (Mr PBI), some others and I, felt that these ideas seemed to make some sense and so I put them here for comment. FWIW, when I see these discussions I'm always left wondering what's the bad part? I do think there are problems, but there doesn't seem to be a clear defined set of what is wrong. IMO, there should be a defined set of goals to judge possible implementations against. Let me start by saying FreeBSD ports is by far the best system I have used to date. but as good as it is, there is room for improvement. Being a FreeBSD user now for many years, one thing I think would be nice is: being able to have easier access to development ports( Masked ports kinda like Gentoo). Masking ports and packages in general introduces all sorts of fun new complexity for end users as well as maintainers. The last time I used Gentoo (which was only a matter of months ago), a lot portage packages were still masked even though they've been stable for months, years, etc. This is very annoying for me as an end-user because bug blah could be fixed in a later release but in order to unmask the pieces for version blah, I had to unmask 10~15 other `unstable packages', which greatly increased the chance of instability on my system (this was particularly the case back several years ago, but Gentoo has become more conservative over the years, and appears to be approaching some level of equilibrium with Fedora, Ubuntu, etc in terms of releases and package versioning). right now is a GREAT example, currently there are new Gnome ,KDE and Xorg. these are all MAJOR ports,dependencies run deeper and deeper with every release. there can never be enough testing...but they all exist in random subversion servers around the web... ports isn't going to solve this. Post the Xorg modularization (which needed to occur anyhow because Xorg and Xfree86 before that was were monolithic beasts), I personally don't see that change in the amount of flux on a quarterly cycle, and the number of packages I install today isn't that much greater than back 6 years ago when I started using FreeBSD. So, while there might be some claim here to note, I think it's mostly exaggerated. I would very much like to help test these Major ports, but installing them is a pain. there should be some sort of overlay system in place, so I can just build the development ports after agreeing to a few well placed warnings of course. and Well if I hose my system all to hell.. well then I could just click on a bunch of PBI's and I am back in business... Ok, apart from the interface (click a PBI, and magically you have packages installed)... how is this really different from binary packages? Have you tried installing binary packages lately via pkg_add? If not, I'd give it a shot instead of installing from ports. yes but there are still dependency problems if you want to install a single package and you installed all the previous ones a year ago. With PBIs each package is self standing, so you can install one and not worry if it requires a different version of some library to what you installed last year. better still, make the development ports a PBI, I am just thinking out loud here,but that may work, toughts? one could say I could use merge scripts like marcusmerge for example, or use Virtualbox... but for large ports like Xorg and gnome or KDE, virtualbox doesn't cut it yet... thinks like Nvidia Video cards, multiple monitors, USB devices, and whatnot do not work on virtual box.. PBI's for development ports, with all the dependencies, wrapped in one package. Ok, well here's the thing. Instead of having N shared dependencies and libraries in /usr/local/lib, you'd have N**2 shared dependencies and libraries in each and every package. Now, let's look at $ ls -l irssi-0.8.14_1.tbz ~/Downloads/Irssi0.8.14_1-PV0.pbi -rw-r--r-- 1 gcooper gcooper 6856203 Apr 10 00:05 /usr/home/gcooper/Downloads/Irssi0.8.14_1-PV0.pbi -rw-r--r-- 1 root wheel 517442 Apr 10 00:07 irssi-0.8.14_1.tbz The .tbz file is a file created with pkg_create -b, and the other file is the PBI I pulled off of http://www.pbidir.com/bt/download/210/2079 . Big difference in size (13.25 fold difference). Yes but that is a worst case thing. We are talking about making a system where the PBIs contain all the libraries needed but that only some of them are installed, when there is not already the same one (i.e. identical) installed by a previous PBI. so if you installed, say, 20 PBI from the same 'set' you woudl only be installing one copy of the libraries that PBIs only comprise a small set of packages in FreeBSD; if my understanding is correct based on a
Re: ports and PBIs
On 4/10/10 3:35 AM, Garrett Cooper wrote: On Sat, Apr 10, 2010 at 1:45 AM, Julian Elischerjul...@elischer.org wrote: On 4/10/10 12:20 AM, Garrett Cooper wrote: On Fri, Apr 9, 2010 at 11:28 PM, Sam Fourman Jr.sfour...@gmail.com wrote: On Fri, Apr 9, 2010 at 10:11 PM, Adam Vande Moreamvandem...@gmail.com wrote: On Fri, Apr 9, 2010 at 8:31 PM, Julian Elischerjul...@elischer.org wrote: Alfred Perlstein , Matt at ix systems Kris (Mr PBI), some others and I, felt that these ideas seemed to make some sense and so I put them here for comment. FWIW, when I see these discussions I'm always left wondering what's the bad part? I do think there are problems, but there doesn't seem to be a clear defined set of what is wrong. IMO, there should be a defined set of goals to judge possible implementations against. Let me start by saying FreeBSD ports is by far the best system I have used to date. but as good as it is, there is room for improvement. Being a FreeBSD user now for many years, one thing I think would be nice is: being able to have easier access to development ports( Masked ports kinda like Gentoo). Masking ports and packages in general introduces all sorts of fun new complexity for end users as well as maintainers. The last time I used Gentoo (which was only a matter of months ago), a lot portage packages were still masked even though they've been stable for months, years, etc. This is very annoying for me as an end-user because bug blah could be fixed in a later release but in order to unmask the pieces for version blah, I had to unmask 10~15 other `unstable packages', which greatly increased the chance of instability on my system (this was particularly the case back several years ago, but Gentoo has become more conservative over the years, and appears to be approaching some level of equilibrium with Fedora, Ubuntu, etc in terms of releases and package versioning). right now is a GREAT example, currently there are new Gnome ,KDE and Xorg. these are all MAJOR ports,dependencies run deeper and deeper with every release. there can never be enough testing...but they all exist in random subversion servers around the web... ports isn't going to solve this. Post the Xorg modularization (which needed to occur anyhow because Xorg and Xfree86 before that was were monolithic beasts), I personally don't see that change in the amount of flux on a quarterly cycle, and the number of packages I install today isn't that much greater than back 6 years ago when I started using FreeBSD. So, while there might be some claim here to note, I think it's mostly exaggerated. I would very much like to help test these Major ports, but installing them is a pain. there should be some sort of overlay system in place, so I can just build the development ports after agreeing to a few well placed warnings of course. and Well if I hose my system all to hell.. well then I could just click on a bunch of PBI's and I am back in business... Ok, apart from the interface (click a PBI, and magically you have packages installed)... how is this really different from binary packages? Have you tried installing binary packages lately via pkg_add? If not, I'd give it a shot instead of installing from ports. yes but there are still dependency problems if you want to install a single package and you installed all the previous ones a year ago. With PBIs each package is self standing, so you can install one and not worry if it requires a different version of some library to what you installed last year. If I'm understanding you correctly you're saying it's an issue when I do: pkg_add A B C # 1 year passes pkg_add D # D depends on A, B, C, of different revisions. pkg_add barfs because it can't find the applications, etc. This is something that's been hashed over a number of times (a few of which I've participated in in #bsdports). There needs to be a simple update command which will handle the action of upgrading packages, because there isn't a proper command that will do so today. Unless PBIs are self-contained entities which have their own sets of dependent utilities and libraries, etc (which you weren't suggesting in the sentence above), or install into a common location with versioned directories (which is a pain in the ass and involves a lot of hardcoded pains dealing with libtool files, libraries, etc -- been there, done that with Gentoo Linux -- there are hack scripts written to work around several possible hardcoded version issue, and there are a handful), AFAIK there's nothing positive and new that PBIs can bring to the table in this regard that can't be implemented in pkg_install as-is, other than the point-click-install user-friendly interface. ok that's your opinion n the matter. I for one think htat hte default ettin for PBIs to install all the dependencies, in this day of 1TB drives, makes sense and is a good capability for us to have, even if not everyone needs to use it. If you can do this
Re: ports and PBIs
On Sat, Apr 10, 2010 at 8:18 AM, k...@pcbsd.org wrote: On Sat 10/04/10 3:35 AM , Garrett Cooper yanef...@gmail.com wrote: [...] yes but there are still dependency problems if you want to install a single package and you installed all the previous ones a year ago. With PBIs each package is self standing, so you can install one and not worry if it requires a different version of some library to what you installed last year. If I'm understanding you correctly you're saying it's an issue when I do: pkg_add A B C # 1 year passes pkg_add D # D depends on A, B, C, of different revisions. pkg_add barfs because it can't find the applications, etc. This is something that's been hashed over a number of times (a few of which I've participated in in #bsdports). There needs to be a simple update command which will handle the action of upgrading packages, because there isn't a proper command that will do so today. Unless PBIs are self-contained entities which have their own sets of dependent utilities and libraries, etc (which you weren't suggesting in the sentence above), or install into a common location with versioned directories (which is a pain in the ass and involves a lot of hardcoded pains dealing with libtool files, libraries, etc -- been there, done that with Gentoo Linux -- there are hack scripts written to work around several possible hardcoded version issue, and there are a handful), AFAIK there's nothing positive and new that PBIs can bring to the table in this regard that can't be implemented in pkg_install as-is, other than the point-click-install user-friendly interface. Incorrect. The difference is in complexity at install-time. Even if you improve the package manager to better resolve and upgrade all related dependencies as a result of doing a firefox upgrade, the fact still remains that just to update one package, you may have to also update a TON of various packages / libraries, any of which may be critical to other applications on your system. If just a single one of those things fails, you can end up breaking a lot of applications on your system or even your entire desktop. PBI system simplifies this process. Updating firefox, since its self-contained, does NOT run the risk of borking anything else on the system. You don't need to work about libpng, libjpeg, or some other seemingly trivial library (to the end user) causing a huge breakage for xorg, or KDE/Gnome, etc. This in my opinion is the fatal flaw of pretty much every open-source system out there right now. Something both windows mac have recognized and dealt with. We instead try to write more and more complex package resolvers, while failing to address the main issue, that with such a complex chain of dependencies for something as simple as upgrading firefox, it increases the chances exponentially that something will break and ruin your day / weekend. PBIs only comprise a small set of packages in FreeBSD; if my understanding is correct based on a mirror referenced in pbidir.com, the number is currently under 500~750 PBIs -- this is drastically smaller than the number of binary packages produced by ports on a regular basis for FreeBSD. solution? well let all the developers develop working ports in progress in one place, give users like me a way to track these changes and install and test them... I think FreeBSD becomes a better place for it. Packages are more of the answer IMO, not PBIs. PBIs are merely a different set of contents and different means of delivering those contents, and while I like the idea of point - click - install, I'm not ready to create unnecessary complexity by having libraries rev'ed according to what the maintainer A believes are correct, even though maintainer B set it differently, and I'm not interested in sacrificing disk space for this reason. If I wanted to use a packaging scheme like this, I should be using Mac OSX as my primary operating system. well no-one is going to make you use PBIs Yes, but if I now have to waste more bandwidth and disk space installing packages, why shouldn't I go to another operating system? Switching over to PBIs will reel in more desktop and entry-level sysadmins, etc, but I fear that it will isolate folks in the embedded market as well as several more seasoned users because of the implications involved with the extra bandwidth requirement and footprint. This gave me a bit of a chuckle. PBI would not be intended as a replacement for ports, rather a utilizing of ports in such a way that we can start building self-contained, stand-alone binaries for end-users and those of us who value their time more than a few MB of disk space. Considering at every BSD conference it seems that the majority of developers are running Mac laptops, it would seem that even some developers agree with the principle, they just aren't doing it on FreeBSD. :) I also, noticed this, and a
Re: ports and PBIs
Garrett Cooper wrote: If I'm understanding you correctly you're saying it's an issue when I do: pkg_add A B C # 1 year passes pkg_add D # D depends on A, B, C, of different revisions. pkg_add barfs because it can't find the applications, etc. This is something that's been hashed over a number of times (a few of which I've participated in in #bsdports). There needs to be a simple update command which will handle the action of upgrading packages, because there isn't a proper command that will do so today. I'm not convinced that the simple update command you mention is actually feasible, much less desirable. (If I want to try out the new Firefox, why does that imply that my year-old Gimp has to be upgraded?) As for feasibility, here's the easy problem: A2.7 requires B3.6 ... one year passes ... A4.8 now requires B7.2 But A4.8 is incompatible with B3.6 and A2.7 is incompatible with B7.2. So neither A nor B can be updated separately without breaking the system. Here's the hard problem: A2.7 requires B3.6 ... one year passes ... I want to install C1.0 which requires B7.2 but there hasn't been a new release of A that works with B7.2. So I now simply cannot have both C1.0 and A2.7 installed at the same time because they require different versions of B. PBI avoids both of these problems. It may be unsuitable for embedded systems[1], but I see no reason we should not extend the existing ports/packages system with additional tools that target certain use cases, and PBI seems a good fit for the desktop case. Tim [1] Actually, PBI might work just fine even for embedded if we address the disk bloat issue. One approach would be to make /Package/Bar/libfoo-2.8.7.so a symlink or hardlink to /Package/Shared/libfoo-2.8.7.so-MD5-hash This gives easy sharing of identical files. It's even easy to handle at install time: * Installer writes libfoo-2.8.7.so to /Package/Shared/libfoo-2.8.7.so-temp-PID of installer * Installer computes hash of file as it's written * Installer renames file (delete if rename fails with EEXIST) * Installer writes symlink or hardlink into /Package/Bar ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On 4/10/10 12:07 PM, Tim Kientzle wrote: Garrett Cooper wrote: If I'm understanding you correctly you're saying it's an issue when I do: pkg_add A B C # 1 year passes pkg_add D # D depends on A, B, C, of different revisions. pkg_add barfs because it can't find the applications, etc. This is something that's been hashed over a number of times (a few of which I've participated in in #bsdports). There needs to be a simple update command which will handle the action of upgrading packages, because there isn't a proper command that will do so today. I'm not convinced that the simple update command you mention is actually feasible, much less desirable. (If I want to try out the new Firefox, why does that imply that my year-old Gimp has to be upgraded?) As for feasibility, here's the easy problem: A2.7 requires B3.6 ... one year passes ... A4.8 now requires B7.2 But A4.8 is incompatible with B3.6 and A2.7 is incompatible with B7.2. So neither A nor B can be updated separately without breaking the system. Here's the hard problem: A2.7 requires B3.6 ... one year passes ... I want to install C1.0 which requires B7.2 but there hasn't been a new release of A that works with B7.2. So I now simply cannot have both C1.0 and A2.7 installed at the same time because they require different versions of B. PBI avoids both of these problems. It may be unsuitable for embedded systems[1], but I see no reason we should not extend the existing ports/packages system with additional tools that target certain use cases, and PBI seems a good fit for the desktop case. Tim [1] Actually, PBI might work just fine even for embedded if we address the disk bloat issue. One approach would be to make /Package/Bar/libfoo-2.8.7.so a symlink or hardlink to /Package/Shared/libfoo-2.8.7.so-MD5-hash This gives easy sharing of identical files. It's even easy to handle at install time: * Installer writes libfoo-2.8.7.so to /Package/Shared/libfoo-2.8.7.so-temp-PID of installer * Installer computes hash of file as it's written * Installer renames file (delete if rename fails with EEXIST) * Installer writes symlink or hardlink into /Package/Bar yeah that's more or less what we were thinking.. hardlinks allow you to garbage collect when the last pbi that needs something is replaced/removed. It's also to handle the cases where library A wants library B. you don't want library A in the shared place looking for B back in the original PBI directory so there may need to be some patching up. ___ freebsd-curr...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
Julian Elischer wrote: On 4/10/10 12:07 PM, Tim Kientzle wrote: [1] Actually, PBI might work just fine even for embedded if we address the disk bloat issue. One approach would be to make /Package/Bar/libfoo-2.8.7.so a symlink or hardlink to /Package/Shared/libfoo-2.8.7.so-MD5-hash This gives easy sharing of identical files. yeah that's more or less what we were thinking.. hardlinks allow you to garbage collect when the last pbi that needs something is replaced/removed. The point of /Package/Shared in this design is basically that it provides a list of all of the files that can be shared, so you avoid doing a full disk search to identify other places that might have this file. You could accomplish the same goal by building and storing a database of sharable files somewhere, of course. (Curiously, no one has mentioned filesystem-level deduping yet as the big hammer solution... ;-) The LD_LIBRARY_PATH issue is the most interesting problem here. I don't immediately see a solution that doesn't include teaching ld-elf.so.1 about some form of per-application library path. Tim ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
Sam Fourman Jr. wrote: I do have a question, assuming PBI's were merged officially into the FreeBSD ports tree, say I had PostgreSQL Server installed, via PBI. then I wanted to tweak a setting so I: cd /usr/ports/databases/postgresql84-server/ make deinstall clean would the PBI at this point be removed? or no because it is self contained? Basically, I believe the proposal here is to add: * make pbi to the ports build system to create a PBI from a port and possibly add * make installpbi * make deinstallpbi to install/deinstall just the resulting PBI. In particular, I don't think anyone is suggesting removing or changing any existing ports/package capability. People who are happy with the existing ports/package system could continue using it exactly as-is. This would imply that you might build Postgres and install it both as a port/package and simultaneously as a PBI. I'm not sure what that would mean, though. The big question, of course: what impact would the addition of make pbi have on existing port/package support efforts? Is this creating extra work for existing maintainers? Should it be optional (enabled per-port) or somehow default? I suspect the next step is for someone to put forward a proposed implementation of make pbi so those interested can start trying it out and see what the impacts are. (If only the GSoC proposal deadline hadn't already passed. ;-) Tim ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
not to be a troll but ... ... for those that want the ease-of-use of PBIs, why not just use PC-BSD in the first place? They seem to have their own QA process in place in terms of keeping the various large applications at a sane level. Kernel development could (just like it is on the Macs) be done in some kind of virtualization context. My own experience with helping people who try to run FreeBSD-CURRENT with an up-to-date ports tree is that there are far too many moving parts for it to be dependable. (For more on how often ports get broken by changes in -CURRENT, see http://wiki.freebsd.org/PortsBrokenOnCurrent. Note that that list is not complete.) mcl ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
[dropped current@ since it doesn't take non-member posts] Tim Kientzle kient...@freebsd.org wrote: The LD_LIBRARY_PATH issue is the most interesting problem here. I don't immediately see a solution that doesn't include teaching ld-elf.so.1 about some form of per-application library path. Maybe install PBI executables with a wrapper of some sort which would set LD_LIBRARY_PATH and exec the real executable, or build similar functionality into a specialized variant of /usr/lib/crt1.o (which BTW seems not to have a manpage)? Nah, that would be too easy. There must be some lethal flaw in it. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On 4/10/10 5:43 PM, per...@pluto.rain.com wrote: [dropped current@ since it doesn't take non-member posts] Tim Kientzlekient...@freebsd.org wrote: The LD_LIBRARY_PATH issue is the most interesting problem here. I don't immediately see a solution that doesn't include teaching ld-elf.so.1 about some form of per-application library path. Maybe install PBI executables with a wrapper of some sort which would set LD_LIBRARY_PATH and exec the real executable, or build similar functionality into a specialized variant of /usr/lib/crt1.o (which BTW seems not to have a manpage)? Nah, that would be too easy. There must be some lethal flaw in it. PBIs already do something like this I believe. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
per...@pluto.rain.com wrote: [dropped current@ since it doesn't take non-member posts] Tim Kientzle kient...@freebsd.org wrote: The LD_LIBRARY_PATH issue is the most interesting problem here. I don't immediately see a solution that doesn't include teaching ld-elf.so.1 about some form of per-application library path. Maybe install PBI executables with a wrapper of some sort which would set LD_LIBRARY_PATH and exec the real executable... PBIs already do something like this, but this doesn't work for setuid/setgid executables. Tim ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On Sat, Apr 10, 2010 at 7:47 PM, Mark Linimon lini...@lonesome.com wrote: not to be a troll but ... ... for those that want the ease-of-use of PBIs, why not just use PC-BSD in the first place? They seem to have their own QA process in place in terms of keeping the various large applications at a sane level. Kernel development could (just like it is on the Macs) be done in some kind of virtualization context. My own experience with helping people who try to run FreeBSD-CURRENT with an up-to-date ports tree is that there are far too many moving parts for it to be dependable. (For more on how often ports get broken by changes in -CURRENT, see http://wiki.freebsd.org/PortsBrokenOnCurrent. Note that that list is not complete.) mcl I have tried PC-BSD . I think it has an important draw back : Its theme is changing its colour cyclically . Any person having chronic illness Vertigo can not endure such continuous colour change . I could not find any place to stop that colour cycling other than to remove PC-BSD and install another operating systems onto its hard disk . In FreeBSD ports system , for me , problem is not the current port system . My idea is to have additional information about ports . For example , when a package is desired to be added by pkg_add , it is downloading the requested package , decompressing it , and saying that it is already installed , and it is not necessary to install it . Instead of this , the following structure ( a more proper one may be suggested , this is only an idea ) may be useful : In the ports , instead of using short names , use after a certain character a signature name of the port : As an example : kde4.version.signature.tbz . In installed systems , always install in directories having that name with signature . When an install is attempted , again use pkg_add kde4 for easiness , not its long name , or kde4.version to select a specified version . pkg_add should compute the signature of the installed port kde4, and check its value to installed signature name . If they are different , the port is destroyed ( install it unconditionally ) , otherwise proper . pkg_add should check port kde4... in ports . If their signatures are the same , it is not necessary to download and install it . For the dependencies , with a port kde4.tbz , maintain a kde4.xml listing all the dependencies , in which they may be inspected recursively ( Such lists are displayed in ports related web pages in www.freebsd.org ) . By checking all the related xml files and installed ports in a system , it will be possible to decide installation possibility of a port attempted to be installed without downloading actual port files . In a directory in the system , maintain a subdirectory of ports : Failed_Builds . Into this directory , store names of the packages which failed during building . When a package is attempted to be build , for itself and its dependencies , check that Failed_Builds directory for matching names . If there exists any one of them , do not start to build , because it will not be successfully completed . ( Entries from that directory may be deleted manually to allow build tries , and successful build may check this directory to remove matching entries if it is present . ) This Failed_Builds list is important , because when that information is not used , the same failed build is attempted many times for an install of some packages . Personally , I am not against an additionally available PBI directory in ports tree . Some users may prefer to use them although some packages will be repeatedly stored in different PBI packages and will be downloaded for each of them . Thank you very much . Mehmet Erol Sanliturk ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On Sat, Apr 10, 2010 at 10:03 AM, Julian Elischer jul...@elischer.org wrote: On 4/10/10 3:35 AM, Garrett Cooper wrote: [...] If I'm understanding you correctly you're saying it's an issue when I do: pkg_add A B C # 1 year passes pkg_add D # D depends on A, B, C, of different revisions. pkg_add barfs because it can't find the applications, etc. This is something that's been hashed over a number of times (a few of which I've participated in in #bsdports). There needs to be a simple update command which will handle the action of upgrading packages, because there isn't a proper command that will do so today. Unless PBIs are self-contained entities which have their own sets of dependent utilities and libraries, etc (which you weren't suggesting in the sentence above), or install into a common location with versioned directories (which is a pain in the ass and involves a lot of hardcoded pains dealing with libtool files, libraries, etc -- been there, done that with Gentoo Linux -- there are hack scripts written to work around several possible hardcoded version issue, and there are a handful), AFAIK there's nothing positive and new that PBIs can bring to the table in this regard that can't be implemented in pkg_install as-is, other than the point-click-install user-friendly interface. ok that's your opinion n the matter. I for one think htat hte default ettin for PBIs to install all the dependencies, in this day of 1TB drives, makes sense and is a good capability for us to have, even if not everyone needs to use it. It's more than just diskspace though. Consider the fact that now you're going to lose a lot of the memory sharing between shared libs and what-not, and now you'd have to be running N number of daemons . Take PCBSD for instance -- do they really revision packages with unshared dependencies, or are all of the dependencies updated at once? This becomes a big issue as you can't have dissimilar applications like dbus, gamin, openssh, etc running on the same system at one time. How does the PBI layout plan to deal with this kind of conflict -- apart from jails, which would greatly increase the required footprint...? If you can do this with package code, Maybe you will supply the packages.. Not quite sure what you meant here. better still, make the development ports a PBI, I am just thinking out loud here,but that may work, toughts? one could say I could use merge scripts like marcusmerge for example, or use Virtualbox... but for large ports like Xorg and gnome or KDE, virtualbox doesn't cut it yet... thinks like Nvidia Video cards, multiple monitors, USB devices, and whatnot do not work on virtual box.. PBI's for development ports, with all the dependencies, wrapped in one package. Ok, well here's the thing. Instead of having N shared dependencies and libraries in /usr/local/lib, you'd have N**2 shared dependencies and libraries in each and every package. Now, let's look at $ ls -l irssi-0.8.14_1.tbz ~/Downloads/Irssi0.8.14_1-PV0.pbi -rw-r--r-- 1 gcooper gcooper 6856203 Apr 10 00:05 /usr/home/gcooper/Downloads/Irssi0.8.14_1-PV0.pbi -rw-r--r-- 1 root wheel 517442 Apr 10 00:07 irssi-0.8.14_1.tbz The .tbz file is a file created with pkg_create -b, and the other file is the PBI I pulled off of http://www.pbidir.com/bt/download/210/2079 . Big difference in size (13.25 fold difference). Yes but that is a worst case thing. We are talking about making a system where the PBIs contain all the libraries needed but that only some of them are installed, when there is not already the same one (i.e. identical) installed by a previous PBI. so if you installed, say, 20 PBI from the same 'set' you woudl only be installing one copy of the libraries that See above comment. PBIs only comprise a small set of packages in FreeBSD; if my understanding is correct based on a mirror referenced in pbidir.com, the number is currently under 500~750 PBIs -- this is drastically smaller than the number of binary packages produced by ports on a regular basis for FreeBSD. solution? well let all the developers develop working ports in progress in one place, give users like me a way to track these changes and install and test them... I think FreeBSD becomes a better place for it. Packages are more of the answer IMO, not PBIs. PBIs are merely a different set of contents and different means of delivering those contents, and while I like the idea of point - click - install, I'm not ready to create unnecessary complexity by having libraries rev'ed according to what the maintainer A believes are correct, even though maintainer B set it differently, and I'm not interested in sacrificing disk space for this reason. If I wanted to use a packaging scheme like this, I should be using Mac OSX as my primary operating system. well no-one is going to make you use PBIs Yes, but if I now have to waste more bandwidth and disk space installing
ports and PBIs
sorry for the cross-post.. Last night at the Bay Area FreeBSD Users Group meeting we had a discussion about ports, and what is good about them and what is bad about them. This has been a topic of discussion quite a bit recently and we were looking for a solution that would allow us to keep the good parts of the current ports system but would allow us to give a better user experience for non guru users. The scheme we came up with involves a merging of the ports tree and the PBI system, developed for PC-BSD. Basically, the addition of a makepbi keyword in the .mk files to allow the automatic generation of PBIs for 'simple' ports such as 'cowsay' (the canonical simple app). More complicated apps would need manual work in Makefile or in a separate pbi-recipe file, but once the support was done we could proceed one port at a time. Not all ports make sense in a PBI format. (e.g. libraries etc. may not) One issue that was raised is the increase of storage overhead when using PBI packages as they include a copy of all required libraries and resources, which means that one would very quickly get duplicate copies of things. Our suggestions include the ability of the PBI management software to resolve and (using hard links) eliminate duplicate items. This is not as easy as it sounds but can be achieved using a special variant of 'objcopy' (at least that is our theory). The aim is to make all apps installed on a system much more resilient to dependency problems. In addition there was discussion on how builds need to be doable as non-root uids sometimes, and that users on a system should be able to install packages (PBIs) as thie selves to get local versions of apps for themselves. Alfred Perlstein , Matt at ix systems Kris (Mr PBI), some others and I, felt that these ideas seemed to make some sense and so I put them here for comment. Julian ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: ports and PBIs
On Fri, Apr 9, 2010 at 8:31 PM, Julian Elischer jul...@elischer.org wrote: Alfred Perlstein , Matt at ix systems Kris (Mr PBI), some others and I, felt that these ideas seemed to make some sense and so I put them here for comment. FWIW, when I see these discussions I'm always left wondering what's the bad part? I do think there are problems, but there doesn't seem to be a clear defined set of what is wrong. IMO, there should be a defined set of goals to judge possible implementations against. In manys ways, simply having a Lastest package archive for RELEASE that is rebuilt say weekly which would resolve most of the problems I run into and address some the non-power users desires as well. Anyways, I'm sure this will be an entertaining thread. -- Adam Vande More ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
