Re: thoughts on architectures
On Wed, Feb 13, 2002 at 02:13:13AM -0500, [EMAIL PROTECTED] wrote: > > I guess I'll have to see about coding support for .note.ABI-tag sometime. > Think they'd accept a patch for that? Probably the best person to contact directly about this is David O'Brien <[EMAIL PROTECTED]> who is the toolchain guy. Or post to <[EMAIL PROTECTED]> although I wouldn't expect an entirely favourable response to the idea of Debian GNU/BSD :-) Tony.
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 06:34:14AM +, Tony Finch wrote: > In article <[EMAIL PROTECTED]> you write: > > > >I need to look into it a bit more, and figure out exactly what FreeBSD does > >and doesn't do with this. > > Everything you need to know is in /usr/src/sys/kern/imgact_elf.c: > > /* We support three types of branding -- (1) the ELF EI_OSABI field >* that SCO added to the ELF spec, (2) FreeBSD 3.x's traditional string >* branding w/in the ELF header, and (3) path of the `interp_path' >* field. We should also look for an ".note.ABI-tag" ELF section now >* in all Linux ELF binaries, FreeBSD 4.1+, and some NetBSD ones. >*/ > > There's a table of the supported ABIs which is scanned when executing an > ELF binary. On the first scan an ABI is found if either the OSABI field > matches (number 1 above) or if the second eight bytes of the ELF header > match (which used to be "FreeBSD" instead of nulls as is usual) (number > 2 above). On the second scan an ABI is found if the file's .interp > section matches (number 3 above). To work correctly with a normal gcc (not hacked like the one in the FreeBSD sources), it needs to do the .note.ABI-tag check. #3 doesn't work for static binaries because they don't have an interpreter, which would be why my test box didn't boot the first time I tried it. I guess I'll have to see about coding support for .note.ABI-tag sometime. Think they'd accept a patch for that?
Re: thoughts on architectures
On Tue, Feb 12, 2002 at 08:38:19AM -0600, Marcus Brinkmann wrote: > On Mon, Feb 11, 2002 at 01:12:28PM -0500, [EMAIL PROTECTED] wrote: > > > > Sure: apt_0.5.4_freebsd-i386.deb vs. apt_0.5.4_i386.deb > > > > Oh that, well. I don't care about the name. Using a human readable name > has advantages, and should bbe preserved if possible. See my other mail, > too. Someone remind me what the problem with a file such as apt_0.5.4_i386_32_netbsd_elf_netbsd-libc.deb is? Seriously - it has all five fields, in a sane order (least -> most specific: processor, mode, OS, binary format, libc); the use of _ in files to designate new information fields is already well established as being a demarcation of meta-data, and files in the old format can be trivially renamed to the new one, if you define what the legacy values are (that is, for Debian, the legacy values would be "32 bit, linux, elf, gnu-libc"). It does mean that some of the non-released architectures (hurd-*, netbsd-*, freebsd-*, and probably sparc64) would have to have their files renamed, which is a perl script that takes about an hour to write *and* check. *If* I understand apt and dpkg correctly, it doesn't even require massive symlink trees, since you just change the Packages files to point to the new names for all arches released with Woody (I'm presuming, here, that this is too major a change to make it in before Woody++); all ports which are released at Woody++ and later would have utilities that understood the new format, and thus, wouldn't need legacy support in the Packages file. -- *** Joel Baker System Administrator - lightbearer.com [EMAIL PROTECTED] http://users.lightbearer.com/lucifer/
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 01:12:28PM -0500, [EMAIL PROTECTED] wrote: > Having distributions for particular architectures wouldn't be required. All > you need is stable, testing and sid. Makes dinstall's job easy. For CD sets, > you pick a kernel, and pull a list of packages that are compatible with it. > The same algorithm dselect or apt would be using. I think I have not provided sufficient rationale about this point. The reason I don't want to drop the concept of a distribution is mostly that: It is not important at all that dinstalls job is easy. Quite the opposite, it is important that what all people use is easy. And most people will use a quite precisely defined architecture, and it would be a waste if everyone would download the information about all packages just to dump most of it. A plus point for your argument would be merging distributions, so if you have two repositories, and some package from one distribution enables packages in the other distribution that you could not install otherwise. Then it is indeed an advantage. But for this type of application I suggest that all tools for creating a pseudo distribution that is the virtual merge of two or more distributions (or parts thereof) are available and not too hard to use. We need such tools anyway for CD creators like Phil, mirror admins and the archive maintainers themself need it probably, too. I think having an additional stage that defines a distribution and cares abouts its consistency helps modularity in the scheme, makes it for most people much easier, and doesn't prevent to do anything fancy you would get with not having it. Thanks, Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 01:12:28PM -0500, [EMAIL PROTECTED] wrote: > > Of course you have to specify this if you want to use a flexible > > architecture system. But it should not be part of the design of the > > architecture system (eg nothing should depend on the selection of such > > interface names). > > Actually, I think you'd want to standardize the naming. I was just trying > to establish a concrete example. I didn't follow through well enough, though. Of course you want to standardise it as soon as you start to use it. But only for yourself (your universe). There is no way you can devise a scheme that fits everyone. So, for Debian, you would want to standardise it. And for Debian, it would be a mistake to make radical change. It would be much better to make a copy of the existing scheme in the new interface, adn then add to it over time. > > Why do you want to replace one evil with another? The existing Provides: > > and Depends: can cope with non-existing package names (called virtual > > packages). Those are exactly what you need for the interfaces like the CPU > > etc. > > True, but that makes it difficult to distinguish between between regular > dependacies and these architecture-type dependacies in software. Which is the right thing to do because all such differences are artificial. That is exactly the core point of my idea. As soon as I accept that there are fundamental differences between such environmental dependencies and normal dependencies (to stick with your naming), I have to reconsider the whole idea. In other words, you are not consequent. Either there are differences, then you should explain them, or there are not, and then I don't understand why you want different fields. > Simple example: you want to be able to quickly find all possible packages that > would work on your system. At that point, it'd be very useful to have a file > that looks like: To find all possible installable packages, you have to take all dependencies into account. This should be done by the distributor as a part of the archive management. As you have to do this for the normal dependencies anyway (it is done currently for normal Debian packages), the same procedures will work if you add the arch interfaces to the normal list. Maybe it is worthwhile to split up the checks into two (for example, it might help error diagnostics or increase performance). I have not thought this through. But I don't see conceptional advantages. > > > 4. Drop keeping metadata in package filenames. Make them just a unique > > > string, assigned when the package is installed into the archive. > > > > I don't understand at all what you are aiming at here. > > > > > That gets rid of the need to add something to the filename to > > > distinguish i386 from sparc packages. > > > > Can you cite an example? > > Sure: apt_0.5.4_freebsd-i386.deb vs. apt_0.5.4_i386.deb > Oh that, well. I don't care about the name. Using a human readable name has advantages, and should bbe preserved if possible. See my other mail, too. > It'd be unnecessary. If you have the dependacies in the Packages file, you > can trivially determine which packages are available to your system in dselect > or apt. Well, my arch-handling text describes a different strategy. I don't think this should be part of dpkg or apt, and I don't think it is trivial in the framework I think of. Thanks, Marcus
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 11:07:18AM +0100, David Schmitt wrote: > On Mon, Feb 11, 2002 at 01:25:51AM -0500, [EMAIL PROTECTED] wrote: > > 4. Drop keeping metadata in package filenames. Make them just a unique > >string, assigned when the package is installed into the archive. > >That gets rid of the need to add something to the filename to > >distinguish i386 from sparc packages. Just use the Packages files, > >and the control fields from the packages themselves. I'd be in > >favour of going further: use one Packages file, and determine > >available packages based on the tags your kernel and libc support. > > Which would lead to the problem, that in the pool/ there would be stuff > like: > > pool/main/p/package/: > > package_Version-1_abcde.deb > package_Version-1_lsahd.deb > package_Version-1_iorzq.deb > package_Version-1_mbmnb.deb > package_Version-1_poiuz.deb > package_Version-1_mjuhb.deb > > > Essentially rendering manual download impossible. Actually, I was thinking of going a bit further than that, and drop the package names and versions from the filenames too. Just think of a squid cache. I don't see manual downloads or parital mirroring as a huge problem. It would simply mean that there'd need to be tools to locate the files. For manual downloads, a CGI that returns a URL for the right file could work. Or possibly just a slightly enhanced apt or similar tool. > This is a question to Marcus Brinkmann: > > How can one distinguish > > package_1.0-1.deb (i686,glibc,mmx) and > package_1.0-1.deb (sparc,netbsd4) > > in the pool (i.e. filesystem)? > > I know, in your doc, you don't explicitly specify an 'encoding' for this > dependency information, but people (as simpleminded as I am) would think > about some extra entry in the Depends: field, which wouldn't help much > with filesystemlayout in the mirrors as Philip Charles mentioned in > regard to partial mirrors and things as pointet out above. That's why I'd like to take those parts out of the filename, and put together tools to handle it. It should be a matter of giving a partial mirror program a pattern you want to mirror with, like "sparc and netbsd," and it generates a list of files to mirror. That's fairly painless, and filesystem layout can be just a matter of keeping directories from getting too many files, and filesystems from overfilling. I maintained a program that tracked about 450,000 files, all of which were had information encoded in the filenames. (Ok, it didn't start that large, but it got there fast.) It was a nightmare until I moved the metadata into a database, and renamed every file to a number. I could then do lookups in the database table, and locate the files much more quickly. (I gained an order of magnitude speed increase in my user interface.) I also was able to balance out the directories. I had some directories with 10,000 files, and other with a few hundred. When I was done, I had 4112 directories, with about 100 files in each. I think a similar approach would be helpful for Debian. Database tables can be indexed, and can handle much more complicated queries than filesystems can. Having information in the filename really doesn't help much of anything except manual downloads, and I think that can be managed.
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 12:47:32PM +0100, Marcus Brinkmann wrote: > On Mon, Feb 11, 2002 at 01:25:51AM -0500, [EMAIL PROTECTED] wrote: > > I think you could say that a binary's environment is made up these things: > > Whatever. The exact set of virtual package names that make up the > architecture depends on your needs (I called this the universe). > > Of course you have to specify this if you want to use a flexible > architecture system. But it should not be part of the design of the > architecture system (eg nothing should depend on the selection of such > interface names). Actually, I think you'd want to standardize the naming. I was just trying to establish a concrete example. I didn't follow through well enough, though. > > 1. Drop the Architecture field. > > 2. Add two new multi-value fields. I'll call them Env-Provides, and > >Env-Depends. Probably someone else can think of better names. > > Why do you want to replace one evil with another? The existing Provides: > and Depends: can cope with non-existing package names (called virtual > packages). Those are exactly what you need for the interfaces like the CPU > etc. True, but that makes it difficult to distinguish between between regular dependacies and these architecture-type dependacies in software. > > 3. Make those fields provide and depend on tags which are completely > >separate from package names. This makes it easy to distinguish > >what packages a package depends on from the environment it's been > >built for. > > I don't understand what you mean here. Can you elaborate? > > > It also makes it easier to make an index of packages > >for a particular architecture combination, > > Why? Simple example: you want to be able to quickly find all possible packages that would work on your system. At that point, it'd be very useful to have a file that looks like: Package: telnet Env-Depends: i386, elf, gnu-libc6 Package: wdiff Env-Depends: alpha, elf, gnu-libc6.1 It's called an index, and it's commonly used in databases to speed things. I think we should also consider using a db3 databases rather than flat text files, but that's a different issue... > > and doesn't pollute the > >package namespace with a lot of virtual packages. > > Beauty is in the eye of the beholder. I think having two fields duplicated > without any real technical advantage is adding to ugliness. The advantage is in making it easier to find things. It'd be easier to tell the difference between packages that could be installed, if we had some other package installed, from those that can't be installed because of architectural dependacies. > > 4. Drop keeping metadata in package filenames. Make them just a unique > >string, assigned when the package is installed into the archive. > > I don't understand at all what you are aiming at here. > > >That gets rid of the need to add something to the filename to > >distinguish i386 from sparc packages. > > Can you cite an example? Sure: apt_0.5.4_freebsd-i386.deb vs. apt_0.5.4_i386.deb If you drop the Architecture concept, you have to find another way to name the files. I'm saying why put any information into the filename? You can get at it with dpkg, anyway, and it's in the Packages files, so why does it need to be in the name? I'm thinking of squid caches. Squid doesn't put any information into filenames, it just load balances across directories, and locates files with its database. I don't see why that wouldn't be a good solution here. > >favour of going further: use one Packages file, and determine > >available packages based on the tags your kernel and libc support. > > I think you misunderstand the purpose of the packages file. The packages > file determines what is in the distribution for a specific architecture. So > dinstall (katie, whatever) would decide if a new package in the pool would > enter a packages file or not based on the dependencies in the package. > > I am probably misunderstanding you, but without an idea want you want to > achieve with 4 above I have trouble to see why you would want to change > that. It'd be unnecessary. If you have the dependacies in the Packages file, you can trivially determine which packages are available to your system in dselect or apt. All you have to do is find dependacies that can't be satisfied without replacing your kernel. And if you install a kernel that supports another sort of binary, you can automatically show more available packages. My suggestion about using new fields rather than Depends was aimed at making that faster and easier to determine, but with a real database, rather than text files, it'd probably be unnecessary. Indexes would solve that nicely. Having distributions for particular architectures wouldn't be required. All you need is stable, testing and sid. Makes dinstall's job eas
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 11:07:18AM +0100, David Schmitt wrote: > Which would lead to the problem, that in the pool/ there would be stuff > like: > > pool/main/p/package/: > > package_Version-1_abcde.deb > package_Version-1_lsahd.deb [...] > Essentially rendering manual download impossible. Which is indeed a problem. > This is a question to Marcus Brinkmann: > > How can one distinguish > > package_1.0-1.deb (i686,glibc,mmx) and > package_1.0-1.deb (sparc,netbsd4) > > in the pool (i.e. filesystem)? > > I know, in your doc, you don't explicitly specify an 'encoding' for this > dependency information, but people (as simpleminded as I am) would think > about some extra entry in the Depends: field, which wouldn't help much > with filesystemlayout in the mirrors as Philip Charles mentioned in > regard to partial mirrors and things as pointet out above. This is a good question. I think the best answer I have is that if you want to keep it transparent and usable by a normal user (eg allow manual download without a lot of tinkering in the packages file), then your best bet is to limit your universe to only allow very constrained overlaps between architectures. In other words, you would normally use the traditional encoding and use special encodings for special cases of overlaps. You don't even need a special case for overlap if one package for GNU/Linux works on the Hurd, for example. Of course, the package name wouldn't tell you that it runs on the Hurd, but the dependencies would. The other possibility is to build symlink trees based on the packages file. We got rid of symlink trees in Debian in favor of the packages pool, but the pool has a similar problem in that it is less transparent for users (they have to know the source name of the package). So this problem would only be hardened. These are just some examples. Please don't think I am doing a fallacy here. Although my proposal is evry broad and flexible, I don't think you should really exploit it in that property. In an actual implementation, you would carefully choose the overlaps where they are useful. An actual implementation would hard code some of the generic parts of the design and limit flexibility in favor for transparency and pragmatics. Thanks, Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 05:18:51PM +1300, Philip Charles wrote: > Maybe there should be several "layers" of Packages files. I don't think the distributor should try to anticipate the needs of individual users of the distribution at this level. If you want to get rid of all emulated packages, just remove the emulator providing package (or just drop its Provides:) and remove everything that is not installable now because of dependencies. This is a fully automated process. (I think you already have such a process fro CD building, so this is nothing new for you). There might be other metadata in the package that shows if a package is a special optimization, or is only available if emulated. Such metadata might be available implicit and only be valid in the universe the package is distributed in (eg depends on the selection of virtual names chosen). I think this is something that could be added to the design at any time, when it is more clear how an implementation would look like. This is indeed an implementation detail. Special casing the selection rules for the distribution, so that some packages would be installed in different Packages files (based on the criteria they were enabled for the distribution) certainly be possible. Thanks, Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 01:25:51AM -0500, [EMAIL PROTECTED] wrote: > I think you could say that a binary's environment is made up these things: Whatever. The exact set of virtual package names that make up the architecture depends on your needs (I called this the universe). Of course you have to specify this if you want to use a flexible architecture system. But it should not be part of the design of the architecture system (eg nothing should depend on the selection of such interface names). > 1. Drop the Architecture field. > 2. Add two new multi-value fields. I'll call them Env-Provides, and > Env-Depends. Probably someone else can think of better names. Why do you want to replace one evil with another? The existing Provides: and Depends: can cope with non-existing package names (called virtual packages). Those are exactly what you need for the interfaces like the CPU etc. > 3. Make those fields provide and depend on tags which are completely > separate from package names. This makes it easy to distinguish > what packages a package depends on from the environment it's been > built for. I don't understand what you mean here. Can you elaborate? > It also makes it easier to make an index of packages > for a particular architecture combination, Why? > and doesn't pollute the > package namespace with a lot of virtual packages. Beauty is in the eye of the beholder. I think having two fields duplicated without any real technical advantage is adding to ugliness. > 4. Drop keeping metadata in package filenames. Make them just a unique > string, assigned when the package is installed into the archive. I don't understand at all what you are aiming at here. > That gets rid of the need to add something to the filename to > distinguish i386 from sparc packages. Can you cite an example? > favour of going further: use one Packages file, and determine > available packages based on the tags your kernel and libc support. I think you misunderstand the purpose of the packages file. The packages file determines what is in the distribution for a specific architecture. So dinstall (katie, whatever) would decide if a new package in the pool would enter a packages file or not based on the dependencies in the package. I am probably misunderstanding you, but without an idea want you want to achieve with 4 above I have trouble to see why you would want to change that. > > Yes, exactly. And you need some differences in the algorithm that decides > > when to compile a package from source for a given architecture (the pool > > might contain a compatible but inferior binary package). > > But that's not a problem for the guy who makes the CD's. No, of course not. Thanks, Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 01:25:51AM -0500, [EMAIL PROTECTED] wrote: > 4. Drop keeping metadata in package filenames. Make them just a unique > string, assigned when the package is installed into the archive. > That gets rid of the need to add something to the filename to > distinguish i386 from sparc packages. Just use the Packages files, > and the control fields from the packages themselves. I'd be in > favour of going further: use one Packages file, and determine > available packages based on the tags your kernel and libc support. Which would lead to the problem, that in the pool/ there would be stuff like: pool/main/p/package/: package_Version-1_abcde.deb package_Version-1_lsahd.deb package_Version-1_iorzq.deb package_Version-1_mbmnb.deb package_Version-1_poiuz.deb package_Version-1_mjuhb.deb Essentially rendering manual download impossible. This is a question to Marcus Brinkmann: How can one distinguish package_1.0-1.deb (i686,glibc,mmx) and package_1.0-1.deb (sparc,netbsd4) in the pool (i.e. filesystem)? I know, in your doc, you don't explicitly specify an 'encoding' for this dependency information, but people (as simpleminded as I am) would think about some extra entry in the Depends: field, which wouldn't help much with filesystemlayout in the mirrors as Philip Charles mentioned in regard to partial mirrors and things as pointet out above. Regards, David -- Signaturen sind wie Frauen. Man findet selten eine Vernuenftige -- gesehen in at.linux pgpj0DaiWgzvF.pgp Description: PGP signature
Re: thoughts on architectures
On Mon, 11 Feb 2002 [EMAIL PROTECTED] wrote: > I'd be in > favour of going further: use one Packages file, and determine > available packages based on the tags your kernel and libc support. > > It's a little radical, but I think it could work well. Ouch. Partial mirrors? I mirror source, i386 and hurd-i386 and as a result have 17 Gb in my archive. How would a partial mirror be maintained with a single packages file? Phil. -- Philip Charles; 39a Paterson Street, Abbotsford, Dunedin, New Zealand +64 3 488 2818Fax +64 3 488 2875Mobile 025 267 9420 [EMAIL PROTECTED] - preferred. [EMAIL PROTECTED] I sell GNU/Linux & GNU/Hurd CDs. See http://www.copyleft.co.nz
re: thoughts on architectures
There's a field in the ELF header called OS/ABI. Readelf -h finds it, and it looks like this: Normal binaries: OS/ABI:UNIX - System V FreeBSD binaries: OS/ABI:UNIX - FreeBSD I need to look into it a bit more, and figure out exactly what FreeBSD does and doesn't do with this. basically, when you set OSABI, you are declaring "this is not standard ELF." tools can no longer assume anything about how it works. actually, given that freebsd's ELF *is* not the SYSV ABI, this is probably the right thing for them to do.. (they have a different (slower -- see -fpcc-struct-return) structure return convention than SYSV.) but all the rest of the world besides HPUX use ELF as it is and shouldn't set ELF_OSABI. this field has a bad name, but there you have it :-)
Re: thoughts on architectures
In article <[EMAIL PROTECTED]> you write: > >I need to look into it a bit more, and figure out exactly what FreeBSD does >and doesn't do with this. Everything you need to know is in /usr/src/sys/kern/imgact_elf.c: /* We support three types of branding -- (1) the ELF EI_OSABI field * that SCO added to the ELF spec, (2) FreeBSD 3.x's traditional string * branding w/in the ELF header, and (3) path of the `interp_path' * field. We should also look for an ".note.ABI-tag" ELF section now * in all Linux ELF binaries, FreeBSD 4.1+, and some NetBSD ones. */ There's a table of the supported ABIs which is scanned when executing an ELF binary. On the first scan an ABI is found if either the OSABI field matches (number 1 above) or if the second eight bytes of the ELF header match (which used to be "FreeBSD" instead of nulls as is usual) (number 2 above). On the second scan an ABI is found if the file's .interp section matches (number 3 above). Tony.
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 03:37:36AM +0100, Marcus Brinkmann wrote: > On Sun, Feb 10, 2002 at 09:28:35PM -0500, [EMAIL PROTECTED] wrote: > > Like libc.so.4 on FreeBSD, soon to be libc.so.5? Not compatible with libc5 > > on > > Linux. It's confusing, but I don't know any good way around it. > > Well, then you have to include the OS-ABI field into the dependency name for > all library dependencies. As the information is easily available from the > binary with objdump, this can be as automatic as the soname. You have to > make such decisions based on the universe you live in. If you build a > distribution that only ever has binaries from a single ABI tag, you don't > need to include it. But if you have different ABI tags you want to support, > you mangle it into the dependency name. I think you could say that a binary's environment is made up these things: 1. processor type (i386, alpha, sparc, etc.) 2. processor mode (64/32) 3. kernel (linux, hurd, whatever) 4. object format (ELF/a.out) 5. object format subtypes or emulations 6. libc That ought to be sufficient to define a binaries requirements. I'd like to see something flexible enough to deal with that much complexity, but not so unwieldy that it can't be made to work. So I'll make my suggestion: 1. Drop the Architecture field. 2. Add two new multi-value fields. I'll call them Env-Provides, and Env-Depends. Probably someone else can think of better names. 3. Make those fields provide and depend on tags which are completely separate from package names. This makes it easy to distinguish what packages a package depends on from the environment it's been built for. It also makes it easier to make an index of packages for a particular architecture combination, and doesn't pollute the package namespace with a lot of virtual packages. 4. Drop keeping metadata in package filenames. Make them just a unique string, assigned when the package is installed into the archive. That gets rid of the need to add something to the filename to distinguish i386 from sparc packages. Just use the Packages files, and the control fields from the packages themselves. I'd be in favour of going further: use one Packages file, and determine available packages based on the tags your kernel and libc support. It's a little radical, but I think it could work well. > > Bear in mind that quite a few systems already support multiple kinds of > > binaries, > > My text was written with that in mind. > > > Also, FreeBSD (and possibly NetBSD as well) uses the ELF OSABI field to mark > > it's binaries. > > The GNU/Hurd does it as well. > > > Of course, dpkg doesn't know that... > > dpkg doesn't know a lot of things ;) > > > That should be correct. A distribution already is just a Package file with > > references to files in the pool. No real change there. The difference would > > seem to be in the generation of the Packages file. > > Yes, exactly. And you need some differences in the algorithm that decides > when to compile a package from source for a given architecture (the pool > might contain a compatible but inferior binary package). But that's not a problem for the guy who makes the CD's.
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 04:41:07AM +0100, Marcus Brinkmann wrote: > On Mon, Feb 11, 2002 at 02:35:50PM +1100, matthew green wrote: > > > >> Also, FreeBSD (and possibly NetBSD as well) uses the ELF OSABI field > > to mark > >> it's binaries. > > > >The GNU/Hurd does it as well. > > > > > > hmm, last i looked hurd used an ELF note, like NetBSD. > > Yes, sorry, I thought that was he meant (and he did :). I did not even know > about a special OSABI field. > > ulysses:/tmp/x# objdump -s -j .note.ABI-tag /bin/bash > > /bin/bash: file format elf32-i386 > > Contents of section .note.ABI-tag: > 8048108 0400 1000 0100 474e5500 GNU. > 8048118 0200 1e00 > ulysses:/tmp/x# objdump -s -j .note.ABI-tag /gnu/bin/bash > > /gnu/bin/bash: file format elf32-i386 > > Contents of section .note.ABI-tag: > 8048100 0400 1000 0100 474e5500 GNU. > 8048110 0100 There's a field in the ELF header called OS/ABI. Readelf -h finds it, and it looks like this: Normal binaries: OS/ABI:UNIX - System V FreeBSD binaries: OS/ABI:UNIX - FreeBSD I need to look into it a bit more, and figure out exactly what FreeBSD does and doesn't do with this.
Re: thoughts on architectures
On Mon, 11 Feb 2002, Marcus Brinkmann wrote: > > That should be correct. A distribution already is just a Package file with > > references to files in the pool. No real change there. The difference would > > seem to be in the generation of the Packages file. > > Yes, exactly. And you need some differences in the algorithm that decides > when to compile a package from source for a given architecture (the pool > might contain a compatible but inferior binary package). > Maybe there should be several "layers" of Packages files. The top layer being a basic packages list used by a simple CD vendor like myself, a second which contains more problematical packages that will run more or less eg, specifically optimised packages for i686. A third layer could contain, for example, those that need some form of emulation. These could be called Packages, PackagesA, PackagesB and suitable modifications made to dpkg. I would imagine that there would have to be manual intervention by the guru when installing from PackagesB. I realise that there would have to be some way of creating these Packages files and that the basic problem has been moved to somewhere else, but as a consumer, this is what I would like to see happen. Phil. -- Philip Charles; 39a Paterson Street, Abbotsford, Dunedin, New Zealand +64 3 488 2818Fax +64 3 488 2875Mobile 025 267 9420 [EMAIL PROTECTED] - preferred. [EMAIL PROTECTED] I sell GNU/Linux & GNU/Hurd CDs. See http://www.copyleft.co.nz
Re: thoughts on architectures
On Mon, Feb 11, 2002 at 02:35:50PM +1100, matthew green wrote: > >> Also, FreeBSD (and possibly NetBSD as well) uses the ELF OSABI field to > mark >> it's binaries. > >The GNU/Hurd does it as well. > > > hmm, last i looked hurd used an ELF note, like NetBSD. Yes, sorry, I thought that was he meant (and he did :). I did not even know about a special OSABI field. ulysses:/tmp/x# objdump -s -j .note.ABI-tag /bin/bash /bin/bash: file format elf32-i386 Contents of section .note.ABI-tag: 8048108 0400 1000 0100 474e5500 GNU. 8048118 0200 1e00 ulysses:/tmp/x# objdump -s -j .note.ABI-tag /gnu/bin/bash /gnu/bin/bash: file format elf32-i386 Contents of section .note.ABI-tag: 8048100 0400 1000 0100 474e5500 GNU. 8048110 0100 Thanks, Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de
re: thoughts on architectures
> Also, FreeBSD (and possibly NetBSD as well) uses the ELF OSABI field to mark > it's binaries. The GNU/Hurd does it as well. hmm, last i looked hurd used an ELF note, like NetBSD.
re: thoughts on architectures
Bear in mind that quite a few systems already support multiple kinds of binaries, and there are more on the way. Sparc, ia64 and x86-64 (not sure if x86-64 is a Debian arch yet) all can do both 64 and 32 bit binaries. I think s390 might too, but I'm not sure. The packaging system needs to know how to deal with the situation. The BSDs can run Linux binaries, and also run on some of the same 64/32 bit platforms. on my netbsd/sparc64 box running a 64 bit kernel, i can run these binaries: - native (elf64_sparc) - elf32 sparc (netbsd) - a.out sparc (netbsd) - a.out sunos - elf32 svr4 - elf64 svr4 i don't think we ever did a.out 64bit so that doesn't count... the libraries for these are located in several places: - /usr/lib - /emul/netbsd32/usr/lib - /emul/aout/usr/lib - /emul/sunos/usr/lib - /emul/svr4 - /emul/svr4 (the latter two because solaris defines different paths 64bit code, something that netbsd hasn't gotten around to working out how to do properly yet.) Also, FreeBSD (and possibly NetBSD as well) uses the ELF OSABI field to mark it's binaries. That's how it knows when to start emulating Linux syscalls. So it's effectively using an altered ELF format on i386, as well as a different libc. But it's capable of supporting libc6 in Linux emulation mode. Of course, dpkg doesn't know that... that would be "FreeBSD abuses the ELF OSABI" but nevermind :-) NetBSD uses an ELF note section, rather than mark their binaries as "not ELF" (which is what setting OSABI means -- bad name yes.)
Re: thoughts on architectures
On Sun, Feb 10, 2002 at 09:28:35PM -0500, [EMAIL PROTECTED] wrote: > Like libc.so.4 on FreeBSD, soon to be libc.so.5? Not compatible with libc5 on > Linux. It's confusing, but I don't know any good way around it. Well, then you have to include the OS-ABI field into the dependency name for all library dependencies. As the information is easily available from the binary with objdump, this can be as automatic as the soname. You have to make such decisions based on the universe you live in. If you build a distribution that only ever has binaries from a single ABI tag, you don't need to include it. But if you have different ABI tags you want to support, you mangle it into the dependency name. > Bear in mind that quite a few systems already support multiple kinds of > binaries, My text was written with that in mind. > Also, FreeBSD (and possibly NetBSD as well) uses the ELF OSABI field to mark > it's binaries. The GNU/Hurd does it as well. > Of course, dpkg doesn't know that... dpkg doesn't know a lot of things ;) > That should be correct. A distribution already is just a Package file with > references to files in the pool. No real change there. The difference would > seem to be in the generation of the Packages file. Yes, exactly. And you need some differences in the algorithm that decides when to compile a package from source for a given architecture (the pool might contain a compatible but inferior binary package). Thanks, Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de
Re: thoughts on architectures
On Sun, Feb 10, 2002 at 10:34:36PM +0100, Marcus Brinkmann wrote: > Hi, > > (I am not subscribed. You might CC me on interesting threads/sub-threads) > > From: [EMAIL PROTECTED] > "The major oversight is that he completely fails to mention libc. And for us, > that remains a big issue. Right now, and for immediately forseeable future, > I'm not going to be using glibc." > > There is nothing special about libc. It Depends on some kernel, cpu and > possibly other stuff, and provides a library api (soname). > > Sonames should be handled completely automatic by the build system (in > Debian they aren't) with optional overrides in case of incompatible libs > with the same soname (don't do that, though). Like libc.so.4 on FreeBSD, soon to be libc.so.5? Not compatible with libc5 on Linux. It's confusing, but I don't know any good way around it. > Programs also might depend on an object format, or you might assume the > object format is common for all programs depending on the same interfaces. > EG, if you assume a common object format, all packages depending on > libc.so.6 are ELF by definition. Otherwise the object format has to be > coded into the dependencies. Bear in mind that quite a few systems already support multiple kinds of binaries, and there are more on the way. Sparc, ia64 and x86-64 (not sure if x86-64 is a Debian arch yet) all can do both 64 and 32 bit binaries. I think s390 might too, but I'm not sure. The packaging system needs to know how to deal with the situation. The BSDs can run Linux binaries, and also run on some of the same 64/32 bit platforms. > A program that depends only on i386, libc.so.6 and optionally elf, will run > on all systems that have libc.so.6, understand the elf object format and can > emulate or provide i386 instructions. That's what I was looking for. Essentially, you're setting up dependacies based on CPU and API's needed, rather than using just architecture. That's a good idea, and I'm in favour of it. However, it won't really help me all that much, because just about everything is going to depend on libc6, and it doesn't work on FreeBSD. Also, FreeBSD (and possibly NetBSD as well) uses the ELF OSABI field to mark it's binaries. That's how it knows when to start emulating Linux syscalls. So it's effectively using an altered ELF format on i386, as well as a different libc. But it's capable of supporting libc6 in Linux emulation mode. Of course, dpkg doesn't know that... > Philipp: > "However, we CD image builders are simple folk with an include/exclude > mentality and not much time for subtle distinctions." > > My proposal shall not worry you. A distribution will look similar to the > distributions that exist in Debian right now. There is no need to change > how a distribution will look to a user. For you and others, it will simply > be a list of packages to pick from, out of a pool. That should be correct. A distribution already is just a Package file with references to files in the pool. No real change there. The difference would seem to be in the generation of the Packages file.
Re: thoughts on architectures
Hi, (I am not subscribed. You might CC me on interesting threads/sub-threads) From: [EMAIL PROTECTED] "The major oversight is that he completely fails to mention libc. And for us, that remains a big issue. Right now, and for immediately forseeable future, I'm not going to be using glibc." There is nothing special about libc. It Depends on some kernel, cpu and possibly other stuff, and provides a library api (soname). Sonames should be handled completely automatic by the build system (in Debian they aren't) with optional overrides in case of incompatible libs with the same soname (don't do that, though). Programs also might depend on an object format, or you might assume the object format is common for all programs depending on the same interfaces. EG, if you assume a common object format, all packages depending on libc.so.6 are ELF by definition. Otherwise the object format has to be coded into the dependencies. A program that depends only on i386, libc.so.6 and optionally elf, will run on all systems that have libc.so.6, understand the elf object format and can emulate or provide i386 instructions. Philipp: "However, we CD image builders are simple folk with an include/exclude mentality and not much time for subtle distinctions." My proposal shall not worry you. A distribution will look similar to the distributions that exist in Debian right now. There is no need to change how a distribution will look to a user. For you and others, it will simply be a list of packages to pick from, out of a pool. Thanks, Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de
Re: thoughts on architectures
On Sun, 3 Feb 2002, Andreas Schuldei wrote: > Markus Brinkmanm (of HURD fame) has written this > http://master.debian.org/~brinkmd/arch-handling.txt > which is mandatory reading for us. > > they (the hurd people) are very interested in working closely > together with us, simply since we together might have enough > weight to make debian make more architecture aware. they are > both highly cooperativ and competent, so we better are nice to > them. (c: > > please give some feedback on that document. Marcus' scheme, or some development of it, is fine for people with access to a full Debian archive. The initial minimal installation of an os is no problem. However, we CD image builders are simple folk with an include/exclude mentality and not much time for subtle distinctions. What is more, we tend to like excluding for space reasons. Eg, should (Linux) kernel source be included on a Hurd CD set? It can be installed, but how useful is it? Kernel source is included in the Hurd CDs as the ftp archive is set up this way. Phil. -- Philip Charles; 39a Paterson Street, Abbotsford, Dunedin, New Zealand +64 3 488 2818Fax +64 3 488 2875Mobile 025 267 9420 [EMAIL PROTECTED] - preferred. [EMAIL PROTECTED] I sell GNU/Linux & GNU/Hurd CDs. See http://www.copyleft.co.nz
Re: thoughts on architectures
On Sun, Feb 03, 2002 at 05:59:28PM +0100, Andreas Schuldei wrote: > Markus Brinkmanm (of HURD fame) has written this > http://master.debian.org/~brinkmd/arch-handling.txt > which is mandatory reading for us. You're right, that's a very good document. He's apparently been thinking along similar lines that I have. There are other problems, like sparc64/32, and ia64 or AMD's 64-bit arch supporting i386, that could be helped by this. The major oversight is that he completely fails to mention libc. And for us, that remains a big issue. Right now, and for immediately forseeable future, I'm not going to be using glibc. Another one is branding binaries. Linux emulation could be quite useful on *BSD and maybe hurd, as well. But it would be really helpful if binutils were altered to do branding. That would help make it possible to install i386 binary packages directly onto freebsd-i386, given the appropriate libraries in /compat/linux. There'd be some other headaches, but it could be done. Might help the Hurd developers, too, if they had it available. > they (the hurd people) are very interested in working closely > together with us, simply since we together might have enough > weight to make debian make more architecture aware. they are > both highly cooperativ and competent, so we better are nice to > them. (c: > > please give some feedback on that document. Excellent. I think we're getting to the point where it makes sense to have those conversations. Perhaps we can all agree to strangle the person who wrote the docs for dpkg in SGML... ;-) ---Nathan
