Re: [gentoo-dev] Init systems portage category
Jesús Guerrero wrote: In my opinion, if we really want to speak about a way to implement that kind of snapshoting, we should start thinking about providing a better integration with lvm, from the root. lvm can take care of the snapshots on a non-expensive way, and it would be relatively easy to implement. However a lot of stuff would need to be re-documented, starting from the handbook, and the init system. LVM snapshotting is extremely wasteful - it has no knowledge of the underlying structure of a filesystem. For example, if I moved all the files around on a fairly full ext3 filesystem, an LVM snapshot would consume the full size of the filesystem. However, a filesystem-level solution wouldn't need to store a single byte of data since nothing actually changed. Also - a snapshot restoration obliterates ALL data on the partition that has changed since the snapshot was taken - so unless the essential files are on a separate partition it won't work out well. LVM snapshots really seem to be a solution to atomic backups - if you unmount, snapshot, and remount a filesystem then you can run a self-consistent backup off of the snapshot with minimal downtime. The wasted space isn't a big deal since the snapshot would be deleted before it grew too far. Finally - I'm not to eager to try out lvm2 again anytime soon - I lost a ton of data when something glitched and wiped out data across multiple lvm partitions. I know that the error must have been in the lvm layer (not md or the filesystem), because when I fscked and repaired one filesystem, another filesystem instantly became hosed (on a separate lvm partition). Somehow the partitions had gotten scrambled together and the fsck was crossing partition boundaries. Plus, dmesg was dumping all kinds of compliants at the md layer about the lvm device trying to access out-of-range clusters. Googling I found a few other reports of similar behavior - it seems extremely rare, but very nasty. Fortunately the most important stuff on my PC was backed up (good planning), but it was still a pain - I lost tons of DVR recordings since I don't back that up (not worth the cost vs the value of the data). Now I just run ext3 on top of md and I haven't had any problems. You're right that btrfs will definitely help. However, being able to create a personal stage1 tarball at will would certainly also be useful, and it wouldn't actually consume much disk space.
Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay
On Mon, Oct 12, 2009 at 04:50:23PM -0400, Mike Frysinger wrote: > what exactly does this "lib32" do ? naming USE flags according to specific > ABI implementations is a bad idea. you have to forget special casing > anything > to "lib32 vs lib64". amd64, while the most common, is hardly extensible. we > must handle multiple ABIs which easily might have the same bitsize. The canonical example for this does still remain MIPS I believe. The most common ABIs in MIPS are: o32, n32 - Both in Gentoo releases n64 - Was experimentally done in Gentoo (default-linux/mips/2007.1-dev/generic-be/n64) o64, eabi, meabi, nubi - Not sure if they were were ever released in any way. Crossdev DOES support the full swath of these last I checked it. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee & Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgpNSvc8hydHP.pgp Description: PGP signature
Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay
On Monday 12 October 2009 15:49:48 Thomas Sachau wrote: > Mike Frysinger schrieb: > > On Sunday 16 August 2009 08:37:37 Thomas Sachau wrote: > >> -for the portage version: It is also in the multilib overlay, but in a > >> different branch called portage-multilib. To use this, you should read > >> the instructions at [1] (doc/portage-multilib-instructions). This one > >> should also mainly work, but there is probably a good amount of packages > >> in the main tree, which may refuse to work with it. > > > > the abi-wrapper doesnt look terribly appealing. why dont we use > > broken/custom -config handling as incentive to convert packages to .pc > > files. pkg-config handles ABI/cross-compile splitting cleanly and > > transparently. > > I am totally unfamiliar with pkg-config, so that would take some time or a > helping hand. going by the pkg-config wrapper we use for cross-compiling, the multilib code should unset: PKG_CONFIG_ALLOW_SYSTEM_CFLAGS PKG_CONFIG_ALLOW_SYSTEM_LIBS then it should export export PKG_CONFIG_LIBDIR to the ABI-specific pkgconfig dir. so it should be something akin to: export PKG_CONFIG_LIBDIR=/usr/$(get_libdir)/pkgconfig any package that adds things to PKG_CONFIG_PATH (grep /etc/env.d/) will be a problem here, but it seems only qt-3 currently falls into this category. the easiest thing will probably be to just disallow this behavior completely. then pkg-config should automatically locate the right abi-specific .pc files and filter the related abi-specific flags. > > bash-4 is stable, so we could start depending on it. > > It still has 3 unstable KEYWORDS including mips. mips doesnt count -- it's moving all to ~arch, and you've counted the bsd arches which are never stable (currently). > > you dont save/restore CPPFLAGS > > Are there any initial values it should get? the current profiles dont set any, but it should be treated as a precious toolchain variable same as any other compiler flag another quick look at _setup_abi_env() looks like it needs work: - LD should not default to `ld` - the -L paths to system dirs in LDFLAGS should not be there -- the toolchain can handle these just fine - missing CPPFLAGS handling - see previous comments re-pkg-config - you dont declare pyver local - the abi if check is uncommon syntax. "! [[ a == b ]]" -> "[[ a != b ]]" > > is there documentation that covers the proposed changes/design to portage > > ? i'm not looking for high level (it runs src_compile twice). i dont > > recall seeing details posted to the portage or gentoo mailing lists ... > > it's hard to review `git diff portage-svn..master`. > > There is currently no documentation for the actual code version. what about documentation for ebuild writers ? only thing i see is how to install the new version of portage. > In short, there are those changes: > > 1. ability to run phases multiple times in ebuild.sh: > > change from > ... > dyn_unpack() > src_unpack > > dyn_compile() > src_compile > > dyn_install() > src_install > ... > > to basicly > > ... > dyn_unpack() > for each ABI > call set_abi > create ABI workdir > set ABI envronment variables > call src_unpack > > dyn_compile() > for each ABI > call set_abi > use workdir for current ABI > set ABI environment variables > call src_compile > > dyn_install() > for each ABI > call set_abi > use workdir for current ABI > set ABI environment variables > call src_install > call _finalize_abi_install > create gentoo-multilib headers (if needed) > ... so basically what we already do in like glibc and sandbox. at the base level, should be fine since it's been working so far with these few ebuilds. how do you control whether the multilib headers are needed ? it'll probably make sense in general, but there are def some packages where this will only get in the way (the toolchain ones). how do you differentiate between packages where multi ABIs make no sense ? for example, most packages that dont install any libraries but just binaries. let's use the simple package app-text/tree. a lot of this multilib code should probably continue to live in the tree as it's a pretty big base of code to formalize that could do with fixes over time. i.e. we figure out that certain paths / define protections dont work so well, so changing them to something else would require PMS changing !? there has been talk before about pushing a lot of basic stuff to the tree so things dont have to be encoded in the PMS. how are packages handled that can only be used via 1 ABI ? any of the packages listed in the amd64 no-multilib package.mask for example. while these are mostly binary-only, this isnt a binary-specific issue. there are a number of packages which only work on x86/ppc but could easily work in a multilib amd64/ppc64 as
Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay
On Tue, Oct 13, 2009 at 1:19 AM, Thomas Sachau wrote: >> the abi-wrapper doesnt look terribly appealing. why dont we use >> broken/custom >> -config handling as incentive to convert packages to .pc files. pkg-config >> handles ABI/cross-compile splitting cleanly and transparently. > > I am totally unfamiliar with pkg-config, so that would take some time or a > helping hand. > You can take a look at http://is.gd/4fSfN for an overview (google cache of original since fdo downtime wiped home directories). I completely agree with Mike that pkg-config makes all this much simpler and consistent. >> bash-4 is stable, so we could start depending on it. > > It still has 3 unstable KEYWORDS including mips. > ~mips ~sparc-fbsd ~x86-fbsd ^^ None of these arches do stable keywords... -- ~Nirbheek Chauhan GNOME+Mozilla Team, Gentoo
Re: [gentoo-dev] Init systems portage category
On Mon, 12 Oct 2009 13:52:49 -0400, Robert Bradbury wrote: > I agree with Wyatt's point. > > Wouldn't there be an easy way to reset the last access date on all of the > files to say 1/1/2009 on a system then execute a relatively robust > multi-user boot (and maybe a world emerge upgrade) and record which files > are actually used during that process, then determine which package they > belong to and label those with some "level of criticality"? In my opinion, if we really want to speak about a way to implement that kind of snapshoting, we should start thinking about providing a better integration with lvm, from the root. lvm can take care of the snapshots on a non-expensive way, and it would be relatively easy to implement. However a lot of stuff would need to be re-documented, starting from the handbook, and the init system. Into my eyes, it's the only serious way to do this at least until btrfs is ready for the masses, and there's a long way until we reach that point still. As for the package bits, it's true that the semantic and delimitation about what's part of the system and what isn't, and the mechanism to handle some things could be better, but I've grown accustomed to the way it is and I really don't care if that changes or not. -- Jesús Guerrero
Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay
Mike Frysinger schrieb: > On Sunday 16 August 2009 08:37:37 Thomas Sachau wrote: >> -for the portage version: It is also in the multilib overlay, but in a >> different branch called portage-multilib. To use this, you should read the >> instructions at [1] (doc/portage-multilib-instructions). This one should >> also mainly work, but there is probably a good amount of packages in the >> main tree, which may refuse to work with it. > > the abi-wrapper doesnt look terribly appealing. why dont we use > broken/custom > -config handling as incentive to convert packages to .pc files. pkg-config > handles ABI/cross-compile splitting cleanly and transparently. I am totally unfamiliar with pkg-config, so that would take some time or a helping hand. > bash-4 is stable, so we could start depending on it. It still has 3 unstable KEYWORDS including mips. > you dont save/restore CPPFLAGS Are there any initial values it should get? > is there documentation that covers the proposed changes/design to portage ? > i'm not looking for high level (it runs src_compile twice). i dont recall > seeing details posted to the portage or gentoo mailing lists ... it's hard > to > review `git diff portage-svn..master`. > -mike There is currently no documentation for the actual code version. In short, there are those changes: 1. ability to run phases multiple times in ebuild.sh: change from ... dyn_unpack() src_unpack dyn_compile() src_compile dyn_install() src_install ... to basicly ... dyn_unpack() for each ABI call set_abi create ABI workdir set ABI envronment variables call src_unpack dyn_compile() for each ABI call set_abi use workdir for current ABI set ABI environment variables call src_compile dyn_install() for each ABI call set_abi use workdir for current ABI set ABI environment variables call src_install call _finalize_abi_install create gentoo-multilib headers (if needed) ... 2. Adding the internal lib32 useflag and usedeps with some workarounds 3. Added bin/auto-multilib.sh, which contains most functions, merged together from the older version kanaka presented ( => http://dev.gentoo.org/~kanaka/auto-multilib/) and the multilib-native.eclass from master branch of multilib overlay. -- Thomas Sachau Gentoo Linux Developer signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Moving real multilib support into main tree portage with request for council decision
Mike Frysinger schrieb: > On Sunday 11 October 2009 06:21:14 Thomas Sachau wrote: >> as announced in a previous mail, i created a fork of portage, which has >> support to create 32bit libs during compile phase for 64bit platforms >> (currently amd64 tested, ppc64 untested). >> >> In short, it does execute every src_* phase twice with keeping a workdir >> for every ABI and saving some default environment vars (like *FLAGS). >> Since the current ABI is the last one in this order, the 64bit phase will >> overwrite everthing from 32bit phase except those parts, which have a >> different install location, so mostly libs, which go in lib32 instead of >> lib64. >> >> A current ebuild and docs for using it are currently in the >> portage-multilib branch of the multilib overlay. > > your git instructions are overly complicated (doc/portage-multilib- > instructions). your two checkouts and .git/config edit are one command: > git checkout -b portage-multilib origin/portage-multilib > > you really should line wrap that file too > both changed >> I asked zmedico about inclusion into the main svn tree of portage, but he >> requested a council ok before he would be accepting it. So this is my >> request for discussion and ok (in tomorrows or the following meeting) from >> the council. > > getting review + approval in a day or two is pretty unreasonable, especially > when info as to what is being changed/proposed isnt well documented. > -mike Thats why my intention was to get it on todays agenda for discussion (and more people looking at it) and possible decision at the following meeting, so the wording may have been unclear. -- Thomas Sachau Gentoo Linux Developer signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Init systems portage category
I agree with Wyatt's point. Wouldn't there be an easy way to reset the last access date on all of the files to say 1/1/2009 on a system then execute a relatively robust multi-user boot (and maybe a world emerge upgrade) and record which files are actually used during that process, then determine which package they belong to and label those with some "level of criticality"? Its probably also true that this list of files could be part of a "critical system backup" that one could just keep around in a bz2 file for fast recoveries (or even as md5sum's to determine when they might have been hosed by disk errors or viruses) -- or is this something that is already done by some of the selinux options? [I've never used selinux so am unsure of everything it does.] R.
Re: [gentoo-dev] Init systems portage category
2009/10/12 Jesús Guerrero > > But there's one... That what the "system" set is about in first place. We > could argue if creating a new category would be any good or not, that's a > different issue. But there's already a list of packages that's considered > critical for a Gentoo system. That's what "system" is, and you will get a > big red waning when trying to uninstall one package belonging to this > category. > My point would be that the selection criteria isn't particularly robust/strict. Iptunnel, df or du for example are not required to the best of my knowledge for system booting or emerges. Dev-lang/python is on the other hand required for emerging (and is not a "sys" package). I'm also not sure that the warnings are strict enough. In order to upgrade a package (util-linux I think) recently I had to unmerge a library package on which it depended but which conflicted with an upgrade to said library. The unmerge of the library package broke either fsck or mount (I can't remember which). Had I tried to reboot before the upgrade was completed there would have been problems. Big "red warnings" are of no use when one is doing semi-automatic-upgrades (and colored encodings are normally disabled when one dumps the emerge output to a file). What is needed is a separate indicator in emerge outputs indicating that an upgrade is potentially "Dangerous" or should require "Manual" intervention. Anyone who is not a full time developer but who wants to maintain a relatively up-to-date Gentoo system (which IMO is its primary advantage over "packaged" releases such as RedHat, Ubuntu, etc.) is going to want to automate the nightly emerges as much as possible such that no user intervention is required. And that probably works 90% of the time. But there are those 5% of emerges that fail "reasonably" and require some intervention (e.g. bug reports) and those 0.1-1% of emerges that fail (or even succeed) with potential problems that could cost the user days. It is that final category (and it isn't every binary produced by a sys* package) that I am suggesting warrants more attention. Robert
Re: [gentoo-dev] Init systems portage category
2009/10/12 Jesús Guerrero > But there's one... That what the "system" set is about in first place. We > could argue if creating a new category would be any good or not, that's a > different issue. But there's already a list of packages that's considered > critical for a Gentoo system. That's what "system" is, and you will get a > big red waning when trying to uninstall one package belonging to this > category. > > Seeing as we understand @system to be "critical for a functional Gentoo system", the phrase "critical packages" may have been poorly chosen for communicating the concept of "things that, should I be cavalier in playing with them, may leave me with a system that is incapable of playing again without intervention from one of those lovely LiveCD things". Nevertheless, there is a class of "packages that I need to watch out for, because they'll make my life miserable in ways X can only dream about and THEN stab me in the kidneys with a rusty javelin if I'm not careful" under discussion that could probably use some action. It's unfortunate that there's no good way of encoding arbitrary semantic metadata about a small set of packages such that it can be leveraged by various sources to achieve this end... Regards, Wyatt
Re: [gentoo-dev] Init systems portage category
On Mon, 12 Oct 2009 12:45:27 -0400, Robert Bradbury wrote: > On Mon, Oct 12, 2009 at 11:39 AM, Victor Ostorga > wrote: > >> >> I don't know the history about init systems category, but obviously is >> necessary to stablish a category into which init systems should live >> happy forever (sys-init ? app-init? foobar?). >> >> > I don't know what you want to call it, "sys-init" perhaps. But it, and a > number of other packages, e.g. sys-apps/util-linux (which includes mount > and > fsck), openrc, bash, udev, etc. belong in a "special" category for > "packages > which could prevent the system from booting or corrupt file systems" if the > emerges do not work perfectly. I get hung up once or twice a year by > semi-auto-emerging a package not realizing that it is a potential > show-stopper that should be closely monitored (or which should require an > immediate system reboot to see if it broke anything). In contrast, you > could break any of the various X libraries, browsers, etc. and still have a > system from which one could fall back/forward. > > Right now one only knows if an emerge is "N"ew or an "U"pgrade with little > indication as to how badly it could go wrong. > > As far as I know there is no "critical packages" list (or class) which > include those that are likely to create much bigger headaches than common > emerge failures (for example this would include all executables used by the > init/openrc processes) which under ideal circumstances would be part of a > single package that could be compiled with a "static" option. But there's one... That what the "system" set is about in first place. We could argue if creating a new category would be any good or not, that's a different issue. But there's already a list of packages that's considered critical for a Gentoo system. That's what "system" is, and you will get a big red waning when trying to uninstall one package belonging to this category. -- Jesús Guerrero
Re: [gentoo-dev] Init systems portage category
On Mon, Oct 12, 2009 at 11:39 AM, Victor Ostorga wrote: > > I don't know the history about init systems category, but obviously is > necessary to stablish a category into which init systems should live > happy forever (sys-init ? app-init? foobar?). > > I don't know what you want to call it, "sys-init" perhaps. But it, and a number of other packages, e.g. sys-apps/util-linux (which includes mount and fsck), openrc, bash, udev, etc. belong in a "special" category for "packages which could prevent the system from booting or corrupt file systems" if the emerges do not work perfectly. I get hung up once or twice a year by semi-auto-emerging a package not realizing that it is a potential show-stopper that should be closely monitored (or which should require an immediate system reboot to see if it broke anything). In contrast, you could break any of the various X libraries, browsers, etc. and still have a system from which one could fall back/forward. Right now one only knows if an emerge is "N"ew or an "U"pgrade with little indication as to how badly it could go wrong. As far as I know there is no "critical packages" list (or class) which include those that are likely to create much bigger headaches than common emerge failures (for example this would include all executables used by the init/openrc processes) which under ideal circumstances would be part of a single package that could be compiled with a "static" option. Robert
[gentoo-dev] Init systems portage category
Lately I have stepeed into bug 216461 "init systems in sys-apps as well
as in sys-process and even app-admin" and was about to moving
sys-process/minit to sys/apps-minit , but stepped into bug 190982
"move sys-process/{minit,runit} and app-admin/jinit to sys-aps" which
is the same and closed WONTFIX in 2009-08-09 .
I don't know the history about init systems category, but obviously is
necessary to stablish a category into which init systems should live
happy forever (sys-init ? app-init? foobar?).
Regards,
Víctor.
[gentoo-dev] Last rites: app-admin/eselect-news
# Ulrich Mueller (11 Oct 2009) # Blocked by its own dependency, therefore no longer installable. # Use the news module of app-admin/eselect-1.2* as replacement. # Masked for removal in 30 days, bug 288560. app-admin/eselect-news
