Tres Melton posted <[EMAIL PROTECTED]>, excerpted below, on Fri, 24 Jun 2005 17:21:33 -0600:
> Good afternoon gentlemen, > > I have a few questions regarding portage and/or emerge that I would > appreciate some help answering. To update my system I: > > esync > emerge -avDut --newuse world > > > Yesterday I did an esync and a number of packages came up one of which > was sudo. I could swear I remember emerging it but when I checked > the /var/lib/portage/world file it wasn't there. So like a good little > puppet I ran "emerge -avDut --newuse app-admin/sudo transcode" (the > previous line was cut from my history so I put it there the way it was > run) and emerged it again. Today I found another similar oddity and > started investigating. When I looked in /var/lib/portage/world the new > stuff wasn't there either. And neither was sudo and I know I installed > it (again) just yesterday. What is going on here? I have reported it > but it was marked invalid since the file wasn't in world. It is hard to > track down so any thoughts would be appreciated. The modification time > did indicate that the file was altered yesterday but it doesn't appear > to have been. Using /usr/lib/portage/bin/regenworld to rebuild my world > file indicates that the world file is wrong. Looking > at /var/log/emerge.log indicates that I started building my system on > March 25 (sounds right -- received new AMD64 around then) and on March > 29 I first emerged sudo and it completed successfully. The next entry > for sudo was yesterday and that indicated success. I do perform > parallel emerges often (but not yesterday) since the file locking issues > were fixed. Is that causing the issue or does anyone have any other > suggestions. It seems you have potentially two issues, (1) a misunderstanding of the world file and what it actually tracks, and (2) a screwed up portage database (not the world file, but /var/db/portage/*). Note the POTENTIALLY in there, it could be that your understanding of world is correct and I'm misreading you, and/or it could be that the portage database is fine and you are misreading events, and nothing is wrong. First, the world file. Note that the world file does *NOT* track EVERYTHING that's merged, only the stuff directly merged from the command line. Packages merged as dependencies of packages on the command line aren't entered in the world file. The idea is that your world file tracks the actual packages you want, not their dependencies, which may change over time. This has a number of implications. First, it's what the --deep (-D) option is all about. An existing package will work just fine with the version of a dependency originally merged to support it, and won't tend to use any new features in newer versions of that dependency anyway. Thus, with just the --update (-u) flag, emerge will only update the stuff you actually want -- the stuff in world (and your system profile, which for the purposes of this discussion, can be considered dependencies of a system virtual that's always presumed to be in world), naturally updating any dependencies necessary to support new versions of the packages in world, but /not/ updating dependencies where your currently merged version is just fine, even if there are new versions of that dependency available. Many folks, particularly those with slower systems, will prefer this, since it means fewer packages have to be merged -- there may be several updates to a dependency that aren't merged because they aren't needed for stuff in your world file, saving that emerge time, between versions actually NEEDED to support new versions of stuff you DO want directly -- that IS in your world file. (Note that this won't cause security issues with old versions either, because in that case they'll be masked, so emerge will unmerge them at the update, and because the dependency is still there, portage will see it needs something to fill it and will merge the latest unmasked version, which should be a version containing the security update. Thus, not updating dependencies under normal circumstances during an update is fine, security-wise.) The --deep option causes emerge to scan dependencies (as well as other packages in the portage database but not in world -- which may be compile-time but not runtime dependencies or may be no longer needed at all -- the ones emerge depclean, see the next paragraph, would unmerge) and update them as well, for those that want a fully updated system, even if it means more emerging than necessary. The second implication triggers when dependencies change. If package foo-1.x that's in your world file depends on package libbar, but you upgrade to package foo-2.x, which no longer depends on libbar, but instead on libbaz, libbaz will of course be merged, but you'll still have libbar around. If nothing else depends on libbar, running emerge depclean will then unmerge it, and any other packages no longer needed, **PROVIDED** they aren't listed in your world file. If all dependencies are listed in your world file, they'll all continue to remain around even when the package that originally required them is long gone. Obviously, this isn't something you want happening, even if you aren't worried about space, because extra merged packages mean more stuff to wade thru when you are trying to find something you WANT, AND more exposure to security issues based on stuff that's hanging around that you don't even NEED. That takes care of the world file, but doesn't answer the now-logical question of where sudo comes in, in that case. First, as stated above, once merged and in the portage database, the --deep switch along with --update will scan for updates of the package anyway. Still, why is it there -- sudu isn't the kind of package that's likely to be a dependency? To check for sure whether it's a dependency of what's in world or not, run emerge depclean --pretend, and see if it's listed for unmerge. If so, you know it's not depended on by anything in world and /should/ be safe to unmerge -- provided your world file contains everything you /do/ want specifically merged (it might not, and in fact might be a bit of a mess at this point, if you've been merging stuff without understanding how the world file works). If you want to unmerge it, you can, otherwise, add it to your world file (echo app-admin/sudo >> /var/lib/portage/world. If it's not listed there and isn't in your world file, it's gotta be a dependency for something. You should be able to find out exactly /what/ by running an emerge --pretend --tree --emptytree world and following the tree. As to how it got merged in the first place if it's not a dependency and never was, perhaps you emerged it at some point with the --oneshot option, which will NOT put the stuff on the command line in the world file? Or, maybe it was (is?) part of your system profile (which as I mentioned above makes them implicit dependencies)? That should cover that aspect of your question, but there's still a loose end to tie up -- regenworld. regenworld isn't always accurate, and may add stuff to your world file that shouldn't be there, or (if your emerge.log is incomplete) miss stuff that /should/ be there. Just because the world file it creates doesn't match the one you had before, doesn't mean the one you had was wrong. That's why you are supposed to make a backup of the world file before running it. The tool is designed to be used in emergencies, when you've deleted or corrupted your world file, to create a replacement that can be used as a starting point. However, it's not recommended that you rely on it, as it may be wrong. The other aspect of your question, and the reason I mentioned a possibly corrupt portage database, is the twice in a row update of sudo. First, it's possible the package /has/ been updated that fast. I know the one update was for a security issue. I haven't run today's sync and update check myself so I don't know if the package has been updated again or not, but it's possible that the first update screwed up and they released another one. A simply comparison of the version your emerge log says you emerged yesterday, with what portage says you have currently and what it wants to merge today, should reveal whether another update /has/ indeed happened, or if it's trying to remerge the exact same version it did yesterday. If it's still saying you have the old version merged, and it's wanting to remerge the same one you did yesterday, your database may be corrupted in some way. If it's a different version, then your db is fine. I can't really give you any help on a corrupt db, but I /can/ say that at one point, due to bad memory causing hard lockups at the time, I apparently crashed in the middle of an update at some point, and for several days, portage kept trying to emerge clean an old version of a package that had been updated for several days, saying it did so successfully each time, but then cleaning up the /same/ entry again, the next time it did its emerge clean cycle. I never had to remerge the package, portage had the emerge recorded successfully. It just somehow failed to get the unmerge of the old package recorded correctly. I guess it eventually took, because the problem went away on its own, after a week or so. > Another question, at what point is a reboot required? I know that > emerging a new kernel requires a reboot (at least to use it) but what > about some of the other packages? Take glibc for instance, almost > everything links to some part of it (except statically linked programs) > so if you update that should you restart? It seems that the file > locking and linking in the kernel should keep the old stuff in memory > pages as long as it is being referenced by something but everything new > should just use the new libraries. Linux is designed so that a reboot is normally not entirely necessary unless you update your kernel and want to run the new one. When system daemons you may be running are updated, the ebuilds generally trigger a restart of the individual daemon, so that isn't normally necessary to do manually. > Problems are going to arise with things like static variables (ones > that have only one copy in memory regardless of the number of times > called). Is there one copy of the variable for the library, one copy of > the variable for each program that links to the library, one copy of the > variable for each instance of each program, or one copy for each user? > > What about dynamically allocated internal data structures? Are they > shared (w/ shared memory) between the two different library versions? > Does each library (old & new) update their own copy blissfully unaware > of the existence of the other? Many libraries exist on a Gentoo (and > other GNU/Linux) system, I only picked glibc here because it is so high > profile. System libraries, particularly glibc, must always exist. The way this is usually accomplished is by creating the new library, then, in a single operation, pointing the symlink at the new version that previously pointed to the old version (using a command such as ln -sf /lib64/libc-2.3.5.so /lib64/libc.so.6). After that, the old version may be removed. To ensure that the new library is being used, you'll then want to restart anything running that links to it, but newly launched applications will automatically load the new version. Current processes that can continue using the previous version of the library (or that don't load it) need not be restarted. Note that just because an app is running does NOT mean it has loaded everything it might need from linked libraries. Sometimes, they are loaded only when called. There is however a switch to the linker available that forces full linking at program launch. This switch is traditionally used for SUID/SGID applications, due to (generally theoretical) security issues with lazy linking and SUID/SGID. Libraries are only shared in as much as they are identical, so if a newly launched app finds an updated version to link against, that memory will not be shared with the older copy in memory. I happen to use that switch here, after reading about it in connection with a portage QA warning (now toned down, again, because it's traditional/theoretical, nothing seriously urgent) having to do with that SUID/SGID thing. LDFLAGS="-Wl,-z,now" The flag does mean apps take a bit longer to load, and with that load, pull functions and libs into memory that I may not actually ever use during this runtime session. However, there are positives as well. In addition to the security thing, it's nice getting any errors about bad links that might otherwise happen when it tried to load that function, up front. Additionally, while initial load time might be a bit longer, it means less wait when I initiate the action that would actually do the load, later. Finally, with a gig of memory, most of which is normally cache anyway, save for the initial launch of the first program that links a library, it's nearly always all or mostly in cache memory anyway, so the slowdown isn't really noticeable in the vast majority of cases. (One exception would seem to be the initial load of KDE. I noticed HUGE improvements in initial load time there, with gcc-4, likely only so pronounced because of all the libraries I was loading then, that normally would wait and be loaded later when actually accessed. Part of this was due to the -fvisibility changes, no largely revoked pending revision and integration into qt first, but it's still faster, it seems to me, than kde-3.3 as compiled with gcc-3.4 was.) > What about the less obvious programs like X, Gnome, Corba/orbit and > others (listed from less to more obscure) that may be used by currently > running programs? What if spam-assassin is the spam filtering backend > for your mail reader. If that is upgraded then do you need to restart > your mail reader or is the spam filter started each time it is needed? > If bash gets upgraded does that mean that everything running from a > script needs restarted? startx runs a script that doesn't finish until > you exit X. Would startx have the rest interpreted with the newer > version of bash of updated from within X? For rebuilds of X or GNOME/KDE/XFCE/whatever, you'll probably want to exit your desktop environment and X, and restart them, but you shouldn't have to reboot, or even restart any of the system daemons and the like. Shell scripts execute from the already loaded copy, if that's what invokes it. If any other program invokes it, it'll be a new launch and thus be the new version. Remember, however, that with shells, the environment matters as much or more than the version of shell running. Keep in mind the differences between exported and unexported shell variables, and login vs regular shells, in terms of what variables are inherited, vs the ones that are not, vs the ones that get reset by the initialization process. With bash, this can be /quite/ complex, because sub-shell instances use the same executable, but get only the exported environment an external program would get, while sourced instances are part of the same instance, executed in place, while executing bash by name uses one set of rules (and files) for initializing, while executing it using the sh alias uses an older more POSIX-like set, and then there's login shell rules vs ordinary shell rules, in terms of the files parsed (.profile, .bash_profile, .bashrc, and the system versions of same, etc). As I said, /quite/ complex. It /usually/ does what one would expect, but when it's not, figuring out just where it's getting the values it gets can sometimes be an "interesting" exercise. <g> At times, in addition to referring to a list of which files get run at initialization under what conditions, I've had to resort to sticking debug echo statements in various locations so I can see what's being run in what order, and/or what value a variable actually has at a specific point. As should be fairly apparent by now, there's few upgrades that really force a reboot, except to load a new kernel (and with the introduction of kexec, even that may be about to change, altho to what extent depends to a large degree on your definition of "reboot"). That said, for the desktop or workstation user that isn't running any multi-week-long processes, it can at times be more convenient to reboot than to restart a list of individual processes manually. One reboot alternative is to switch init levels, down to init 1 to stop nearly everything, then back to init 3 (or whatever) to restart most of the system. Note that on Gentoo, unlike on many distributions and *ix versions, init 1 does NOT mean single user mode that shuts down all the virtual terminals and what may be running from them. Thus, it's possible to keep a long-running process alive either in another virtual terminal (or in the background in your current terminal), or add them to a couple custom init levels, one with your usual other system stuff in it as well, one without most of it, then switch between those levels, so the long-running process doesn't stop. > I was reading the minutes of the gcc summit and it seems that the > register allocator is going to get reworked again. I saw some of the same news, but that's /waayyy/ beyond me. I very quickly realized I was hopelessly lost and hadn't the foggiest idea what they were talking about, or what the effect on the real world was going to be, even tho the individual words continued to make sense. <g> However, I'm sure there'll be many writeups on it long before any such changes affect me, so I'm not all that worried. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman in http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html -- [email protected] mailing list
