Re: YAPIB (was: Drawing graphics on terminal)
On Fri, Jun 20, 2003 at 10:37:58AM -0700, Tim Kientzle wrote: Yes, meanwhile, the server providing B times out your connection, the whole install gets rolled back, and you have to start again from scratch. Not pretty. Quite. Unless you ship all dependancies as part of the package in the same file, which kind of defeats the object of having packages. Hmmm.. There are two problems here: The first is maintenance. Suppose a couple of friends of mine set up a site with packages that they're building. I want to be able to add their site to my personal list of package sources without having to go bug the Official FreeBSD FTP Package Uber-Person to get their packages added to the master file. This means that my pkg_add needs to be able to search multiple sites no matter what. Fair enough. So, if I state pkg_add -r apache it's going to grab the file from the main site, or the local DB for apache. If I were to specify pkg_add -r ftp://some.site.somewhere.com/~myfriend/pkgs/apache.tar.bz2; or whatever, then obviously I'm not going to look at the local DB for apache, or try and get the re-direct file off of the FreeBSD server - I'm going straight for the pkg I've specified. Don't rely on a single definitive source of package information. Having some sort of redirect support in the INDEX file is fine and easy to add, but you still need the client to be able to search multiple sources. This is one thing the Debian folks got right. I think what I'm talking about allows for that. It allows for a big local database, a remote database maintained by FBSD folks, a smaller footprint than the current packages take AND allows for installs of remote or local packages that are not part of that schema. In addition, it'll also try and build the port if it can't find the pkg. The other problem is that the current -r is fundamentally limited (to a single network source) and draws a rather pointless distinction (you get to search either disk sources with PKG_PATH _OR_ you get to search a network source, but not both). I'd like to erase that distinction so that pkg_add just searches all available sources. That's what I'm saying. -r no longer grabs the package from the remote server, it grabs a file from the remote server that tells you where the packages may be, and failing that, what the port is. Equally, without -r it searches the local database, unless I specifically name a file in the form ftp://blah.com/package.tar.bz2 or ~/mypackage.tar.bz2 or whatever, but if I just pkg_add apache it'll search local DBs in the form similar to my last e-mail. Easy to do, but I'd be cautious with this. Building OpenOffice or KDE from ports is an adventure that most people would rather skip, and pkg_add shouldn't be automatically starting a port compile just because it notices that there's a 1.0.3 port and a 1.0.2 package. Yup. But we can alter the files for those special cases where we know a port takes 14 hours to build, 2Gb of disk space and 512Mb RAM, and say are you REALLY sure you want to do this?. Besides, as far as the example with the plane is concerned, within 6 months, we'll all have 10Mb/sec to our seats, surely? :-) Of course, there's also some merit to working on this issue from the other side. In many cases, port requirements could easily be satisfied from packages. (How many people really need to compile the JPEG library?) Sam Smith e-mailed me over the weekend with a patch he made to the OpenBSD ports where you could specify FROM_PACKAGES or FROM_SOURCE either on the command line with make, or in the conf file and then ports kind of does what we're talking about. So, if by default you have in mk.conf (or whatever it is, memory is poor on Monday mornings) you have FROM_PACKAGES=yes when you do cd /usr/ports/graphics/jpeg; make install it tries to find a package before it builds the source. This is sensible. All of this already exists. Ports already register with the /var/db/pkg DB and the ports framework already has make targets to build packages from any port. Yeah, I'd forgotten about the package command on ports. Never used it, but kind of remembered from the man page, which I've just refreshed my memory of. Because of this though, there is no excuse for a port to be ahead of a package. It wouldn't take much to cvs the ports collection on a daily basis, track those ports that are added or updated and automatically build a package. This already exists; it's called /usr/ports. See the pkg-* files, the Makefile, etc. Those already exist, and can be mined by other tools. (Look at crunchgen for some tips on how to effectively mine data from conforming Makefiles.) No, no, no, no. ports is not what I'm talking about. I know it seems daft, but I think I've lost you somewhere here. Maybe I can get what I want from taking Sam's patch and adapting it for FreeBSD, but it's still not quite what I'm talking about. Let me do some work,
Re: YAPIB (was: Drawing graphics on terminal)
On Thu, Jun 19, 2003 at 12:11:26PM -0700, Tim Kientzle wrote: That would seem to be the hard part. I presume you've looked at SUSE's YAST, Debian's APT, and other such tools? *nods* - nice basis, but not... well... you know. What I have now works as follows: * Start reading the package, extract the packing list, parse it. * Identify any requirements, recursively install those. * Continue reading/installing this package. To clarfiy your meaning of recursiveness in this context (I can think of another way of doing it recursively and I don't have time to peek at code right now), if I want to install package A which requires B and C, B requires D and E, and D requires F, your installer would go Start A - I need B - Start B - I need D - Start D - I need F - Install F - Install D - I need E - Install E - Install B - Install C This has a big problem with network installs. In particular, the download gets stalled while the requirements are added. Over a slow connection, this could leave you with a stalled TCP connection for hours. Not good. In the chain above, if F isn't available for some reason, you have A, B and D all half installed on your machine waiting for a 32Kb library on an un-mirrored FTP server in bulgaria... hmmm... One way to address this would be to separate install-time requirements from run-time requirements. The ports framework already has some concept of this (separates build-time from run-time), but it doesn't seem quite sufficient. If you need it at run time, surely it make sense to grab it at build time? I'm not sure I can see the benefit of seperating it out - you're just going to create a sense of not knowing whether your application is ready to go or not because it's installed but doesn't have the kit it needs around it to make it actually *work* h... I'm looking at a couple of approaches. One is to eliminate -r and instead have a simple list of package sources that get inspected. Debian's package management does something similar to this. For example, you might have an /etc/pkg.conf that specifies the following sources: . /usr/packages/distfiles /nfs-server/packages/distfiles ftp://ftp3.freebsd.org/some/path/packages-5.8-release/ ftp://ftp.joesfreebsdsite.org/some/path/packages-5.8-release/ cdrom:FreeBSD 5.8 CDROM #2:/cdrom/packages Yup, that's what I was thinking, but you would have such a file for each package, thereby meaning packages can live all over the place. In addition, you wouldn't need that file on the local machine, and for backward compatability, the -r switch grabs a file with that info off of the mirrors, the same way the actual packages are now. It means that in 3-4 years when people are no longer trying to do package management with the current stuff, the mirrors *could* reclaim some disk space. This is likely to be an issue if we want to try and get as much stuff out there as possible run up as pkg's. installed. In particular, note that this should allow us to support the CD-ROMs more efficiently, by locating packages on particular CD-ROMs and then prompting the user to insert the correct CD. There is a minor issue here, around the way I'm planning on helping out the OEM/release engineering stuff as part of the installer effort, in that the package might not be on FreeBSD 5.8 CDROM #2 but rather on Dell OEM FreeBSD 6.2 Disk 1, but that's my problem. The more I think about it, the less of an issue it becomes, as I've just written some code in my head around building the release disks that sorts some of this out, but it's an extra req. Note that this is simpler than having some form of master redirect file, since each repository only needs to track what it provides, not what other repositories might offer. Users can mix and match repositories as needed. I'm thinking about backward compatability on the command line for -r that grabs the master re-direct file in the format above.. No opinion on this one. Perhaps you could formulate a couple of scenarios in which it would be beneficial to be able to mix and match these two? Where the port exists, but a pre-built binary isn't available, or where somebody wants easy install with his own configure options. So, in your file above, but where you're explicitly discussing a specific package rather than packages in general, you could have a line, for example: ports:/usr/ports/misc/screen:--prefix=/usr/local with command line switches to force the ports option, pass extra args to configure, etc. This means as and administrator you have one 'place' to look after third-party code, you get the advantage of being able to wrap ports into the /var/db/pkg DB, and if the binaries don't exist you can go back to building from scratch. In fact, with a bit more work, you could write a switch to make a package that can then go onto a mirror with nothing more than a command-line switch, based on the port. It also means that if somebody
Re: YAPIB (was: Drawing graphics on terminal)
On Thu, Jun 19, 2003 at 02:26:13PM -0400, The Anarcat wrote: It's right now to the point I wouldn't consider writing more code for libh, but I'd reuse the ideas in a smaller, plugin-based, swig-foobar rewrite. I went back and re-read the notes on the website about libh's design yesterday, first time in a while. It has some good ideas. Definitely lots to pick up on. So yes, libh is kinda bad, for me at least. And it involves a lot of C++ magic I don't really like. *shudders*. I was sorting out my flat last night getting ready to move, and I found my old undergrad copy of the second edition KR book. I hugged it. C++ is for girls. :-) But, as always, if someone feels like picking the horse (wether it's alive or dead I'm never too sure), feel free! The code is there for anyone who dares to look. I've also put up a doxygen framework and there's a bit of design doc, so no, libh's still no so bad since so much people put so much time in it during its long existence. Cool. I apologize again for being so brutal to this project which had so high hopes and also to all the people who worked on it. It's understandable. Think of it this way - at least the next-gen installer is now likely to get a better name, whoever it is that writes it. :-) -- Paul Robinson ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: YAPIB (was: Drawing graphics on terminal)
Paul Robinson wrote: ... if I want to install package A which requires B and C, B requires D and E, and D requires F, your installer would go Start A - I need B - Start B - I need D - Start D - I need F - Install F - Install D - I need E - Install E - Install B - Install C In the chain above, if F isn't available for some reason, you have A, B and D all half installed on your machine waiting for a 32Kb library on an un-mirrored FTP server in bulgaria... hmmm... Yes, meanwhile, the server providing B times out your connection, the whole install gets rolled back, and you have to start again from scratch. Not pretty. One way to address this would be to separate install-time requirements from run-time requirements. If you need it at run time, surely it make sense to grab it at build time? I'm not sure I can see the benefit of seperating it out The benefit being that only install time requirements actually need to be installed recursively. Run-time requirements can be queued up and installed afterwards. This reduces the likelihood that network installs will get stalled for long periods of time. Of course, this doesn't really solve the underlying problem. As I said, if you can get requirements information from somewhere else (the INDEX files available on CDs and FTP sites), then you can build a full, properly sorted list of packages and install them in order without any stalls. That's the approach I'm planning to pursue next. I'm thinking about backward compatability on the command line for -r that grabs the master re-direct file in the format above.. Hmmm.. There are two problems here: The first is maintenance. Suppose a couple of friends of mine set up a site with packages that they're building. I want to be able to add their site to my personal list of package sources without having to go bug the Official FreeBSD FTP Package Uber-Person to get their packages added to the master file. This means that my pkg_add needs to be able to search multiple sites no matter what. Don't rely on a single definitive source of package information. Having some sort of redirect support in the INDEX file is fine and easy to add, but you still need the client to be able to search multiple sources. This is one thing the Debian folks got right. The other problem is that the current -r is fundamentally limited (to a single network source) and draws a rather pointless distinction (you get to search either disk sources with PKG_PATH _OR_ you get to search a network source, but not both). I'd like to erase that distinction so that pkg_add just searches all available sources. I can see where a flag to inhibit network downloads might be useful. (I'm sitting on an airplane with my laptop and I _think_ I downloaded everything I needed before I left the office.) However, flags like -r that select individual sources just strike me as rather pointless. Perhaps you could formulate a couple of scenarios in which it would be beneficial to be able to mix and match these two? Where the port exists, but a pre-built binary isn't available, Okay, I can see some merit in having pkg_add mine the ports system as a source of packages. Basically, if the ports version is the newest, provide the user an option of installing from there. Easy to do, but I'd be cautious with this. Building OpenOffice or KDE from ports is an adventure that most people would rather skip, and pkg_add shouldn't be automatically starting a port compile just because it notices that there's a 1.0.3 port and a 1.0.2 package. Of course, there's also some merit to working on this issue from the other side. In many cases, port requirements could easily be satisfied from packages. (How many people really need to compile the JPEG library?) ... or where somebody wants easy install with his own configure options. If people really want their own config options, then they should work with the port directly. There's no feasible way to wrap every possible port option into a single tool. The whole point of the ports framework is the extreme flexibility it provides. ... you get the advantage of being able to wrap ports into the /var/db/pkg DB ... you could write a switch to make a package ... based on the port. All of this already exists. Ports already register with the /var/db/pkg DB and the ports framework already has make targets to build packages from any port. ... whereas I'm thinking of a DB on disk, or retrieved over the network for EACH PACKAGE, ... This already exists; it's called /usr/ports. See the pkg-* files, the Makefile, etc. Those already exist, and can be mined by other tools. (Look at crunchgen for some tips on how to effectively mine data from conforming Makefiles.) Tim Kientzle ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: YAPIB (was: Drawing graphics on terminal)
On Wed, Jun 18, 2003 at 11:40:13AM -0400, The Anarcat wrote: - Whether the installer is graphical or not is not the issue. Grey boxes on a blue background with yellow, red and black text is just plain ugly to a society that understands art and interior design. I know you're limited on pallet due to the restrictions of the console, but you can make sysinstall nicer just by changing the colour scheme. You can make it a hell of a lot nicer by making it consistent and functionally useful. Who cares really... Are you going to code it? Are you going to rewrite sysinstall and provide support or are you going to rewrite dialog? You're quoting me there. The answer to your question Are you going to code it? is Yes. Are you going to reqrite sysinstall and provide support or are you going to rewrite dialog? - neither. I'm fed up of it sitting there too. I'm sick and tired of the discussions. So, after I've moved house this month, got my new machine sorted, and then I'll be sitting down to work on it. I'll do it my way, make it a port, and if others like it, I'll make it easy for them to use in building their own ISOs. I do not see a problem with this. As to what I'm writing, well, I'm going to do the design in about four weeks time, and anybody who is interested can take a look. An announcement will probably go up on -hackers and -libh... I'm getting *really* tired of random people popping up on mailing lists and saying what should and shouldn't be done and complaining about how sysinstall sucks. Yes it sucks! So what? What are you going to do about it? See above. I suggest you folks to stop talking and start designing. If someone can come with a clean design of a new package manager/installer, then *maybe* something could come out of it, but in all the time I've been interested in package management in BSD, all the talk I've seen has been moot (e.g. I want a GTK installer! I want a pkgAPI!), without any actual code or design. Just talk impeding progress. I want something that works. To be honest, just something that abstracts /usr/ports and makes use of the pkg-descr files would be more useful than the current blank void navigated with cd and more... The only real improvement I'm aware of is portupgrade, which is doing an extraordinaire job considering the architechture, and just popped up without prior useless, endless, close-minded discussions. Let us not forget Colin's binary upgrade software which looks increasingly interesting. Yeah, noted. Good stuff. I'm just looking at putting a friedndly abstraction over that. libh's dead, folks. It's been dead for a good while now. I was just kicking it to make it look like we could tear something out of this monster. It's not *that* bad is it? :-) -- Paul Robinson ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: YAPIB (was: Drawing graphics on terminal)
Paul Robinson wrote: As to what I'm writing, well, I'm going to do the design in about four weeks time, and anybody who is interested can take a look. An announcement will probably go up on -hackers and -libh... I want something that works. To be honest, just something that abstracts /usr/ports and makes use of the pkg-descr files would be more useful than the current blank void navigated with cd and more... Paul, When you get ready to do some work, let me know. I've been rewriting the guts of pkg_add for the last month or two. I'm pretty pleased with the results so far, but there's still a lot of code to write. So far: * libtarfile works. This is a library that provides simple iteration over tarfiles. It handles format detection (e.g., both old/Posix/GNU formats and gzip/bzip2/etc compression), can 'extract' entries to disk or to an in-memory buffer, etc. The read support is pretty solid; the write support is just a sketch. * Direct package extraction works. I can open a package file from stdin, disk, ftp, etc, and just install it without having to create a temp directory or any of that nonsense. The idea: extract the packing list into memory, parse it, use it to direct the extraction of the rest of the package. This is _MUCH_ faster than the older pkg_add code. * I've also completely overhauled the packing-list parsing code and a lot of the other basic operations. Next steps: * Requirements handling: I have some recursive requirements handling, but I'm not entirely happy with it. I'm exploring other approaches. * Locating packages. This will probably involve building a DB file in /var/db/pkg to record information about what packages are available from which ftp sites, etc. * Handling conflicts gracefully. This may well involve building a DB file that maps filenames-package names so that an attempt to overwrite a file can be immediately tracked back to the conflicting package. * Building a useful library. I'm being careful to keep code as generic as possible, so it should be pretty simple to put a lot of the useful pieces (packing-list management, locating packages, etc) into a library. Like I said, let me know when you're ready to work on this. My stuff is still pretty rough in some spots, but a lot of it should prove useful to anyone working on install issues. Tim Kientzle ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: YAPIB (was: Drawing graphics on terminal)
On Thu, Jun 19, 2003 at 09:42:20AM -0700, Tim Kientzle wrote: When you get ready to do some work, let me know. I've Will do. Just need to move, get DSL installed, and clear down some work and I'm looking at a year of clear weekends and evenings. And this is biting me up inside now... :-) * libtarfile works. This is a library that provides simple iteration over tarfiles. It handles format detection (e.g., both old/Posix/GNU formats and gzip/bzip2/etc compression), can 'extract' entries to disk or to an in-memory buffer, etc. The read support is pretty solid; the write support is just a sketch. v. cool. The other night when I was sitting in the pub sketching on beer mats and cigarette packets to work out what needed doing, one of the first things that sprung to mind was the ability to look inside, if not packages, at least tar files in memory just to get the juicy info out to present the user with choices as to whether that pkg is something they really want, and if so, not have to play around in /tmp or wherever... * Direct package extraction works. I can open a package file from stdin, disk, ftp, etc, and just install it without having to create a temp directory or any of that nonsense. The idea: extract the packing list into memory, parse it, use it to direct the extraction of the rest of the package. This is _MUCH_ faster than the older pkg_add code. Hah. OK, item number two on the list is struck off. Cool. Looks like I'll be writing a GUI then? :-) * Requirements handling: I have some recursive requirements handling, but I'm not entirely happy with it. I'm exploring other approaches. I'd be interested to know why you're not happy with it - is it the concept of recursiveness over a set of requirements in general you think is flawed, or is it your implementation which is niggling you? Requirements and dependancy tracking/handling are big issues. I've resorted to reading papers on how people have approached it in the past and have some vague ideas, and recursiveness has problems. Recursiveness is always problematic though, no matter what domain you take it too - branching is where I was thinking. * Locating packages. This will probably involve building a DB file in /var/db/pkg to record information about what packages are available from which ftp sites, etc. Instead of retrieving the binary when you do a pkg_add -r perhaps you could end up grabbing a file that described where it could *really* be found (thereby alleviating mirrors from carrying GBs of packages, just the descriptions of where they are), including possible locations on disk, and ultimately, ports ... nothing stopping you shipping this db as part of the base install, or like ports is now, but then we get to crux: Shouldn't pkg_add be an abstraction of ports, with the advantage of grabbing a pre-built binary if it's available, with a command line switch to force a traditional cd /usr/ports/category/package; make install clean ??? At the moment we have ports, and we have packages. Maybe it's time to merge to the benefit of both? hides under flameproof blanket snip Like I said, let me know when you're ready to work on this. My stuff is still pretty rough in some spots, but a lot of it should prove useful to anyone working on install issues. Yeah, certainly, thanks for the heads up. I'm sure I would have been hassling you in a fortnight anyway. :-) -- Paul Robinson ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
YAPIB (was: Drawing graphics on terminal)
(or Yet Another Package Installer Bikeshed) [libh CC'd, for the archives] On mer jun 18, 2003 at 06:23:42 +0300, Samy Al Bahra wrote: - Whether the installer is graphical or not is not the issue. Grey boxes on a blue background with yellow, red and black text is just plain ugly to a society that understands art and interior design. I know you're limited on pallet due to the restrictions of the console, but you can make sysinstall nicer just by changing the colour scheme. You can make it a hell of a lot nicer by making it consistent and functionally useful. Who cares really... Are you going to code it? Are you going to rewrite sysinstall and provide support or are you going to rewrite dialog? I'm getting *really* tired of random people popping up on mailing lists and saying what should and shouldn't be done and complaining about how sysinstall sucks. Yes it sucks! So what? What are you going to do about it? Just as YaST, a libsysinstall can be provided to provide a standard UI API for applications to use + various UI plugins (ncurses, QT, GTK, Xaw, you name it) and configuration modules (users, network, ports, etc...). Though, before we all get excited about the possibilities of such an installer, what's happening with libh? Isn't it supposed to deal with all of sysintall's short-comings? All I see now is a lot of talk and no code, maybe such discussion should go to libh's mailing list (where we can talk design there)? Nothing's happening with libh. I suggest you folks to stop talking and start designing. If someone can come with a clean design of a new package manager/installer, then *maybe* something could come out of it, but in all the time I've been interested in package management in BSD, all the talk I've seen has been moot (e.g. I want a GTK installer! I want a pkgAPI!), without any actual code or design. Just talk impeding progress. The only real improvement I'm aware of is portupgrade, which is doing an extraordinaire job considering the architechture, and just popped up without prior useless, endless, close-minded discussions. Let us not forget Colin's binary upgrade software which looks increasingly interesting. libh's dead, folks. It's been dead for a good while now. I was just kicking it to make it look like we could tear something out of this monster. Now back to your scheduled bikeshed. A. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: YAPIB (was: Drawing graphics on terminal)
Sorry to hear you say that. It was probably the only effort (which attempted to solve the larger set of issues and not simply peck away at the problem piecemeal) to ever have any code associated with it. On Wednesday, June 18, 2003, at 08:40 AM, The Anarcat wrote: libh's dead, folks. It's been dead for a good while now. I was just kicking it to make it look like we could tear something out of this monster. -- Jordan K. Hubbard Engineering Manager, BSD technology group Apple Computer ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: YAPIB (was: Drawing graphics on terminal)
I am still doing work on it... but my normal job has been getting in the way for a while. Max Jordan K Hubbard [EMAIL PROTECTED] writes: Sorry to hear you say that. It was probably the only effort (which attempted to solve the larger set of issues and not simply peck away at the problem piecemeal) to ever have any code associated with it. On Wednesday, June 18, 2003, at 08:40 AM, The Anarcat wrote: libh's dead, folks. It's been dead for a good while now. I was just kicking it to make it look like we could tear something out of this monster. -- Jordan K. Hubbard Engineering Manager, BSD technology group Apple Computer ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-libh To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: YAPIB (was: Drawing graphics on terminal)
Max Okumoto wrote: I am still doing work on it... but my normal job has been getting in the way for a while. Max I'm sorry Max. I guess I should have used a bit more diplomacy. But the way I see it, libh was dead even before you got in, the same way it was almost dead before I got in. You've done an awesome job considering the circumstances, but I'm out, for now. ;) No use pretending otherwise. Cheers, A. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
