On Sat, 2005-06-25 at 07:21 -0700, Duncan wrote: > It seems you have potentially two issues, (1) a misunderstanding of the > world file and what it actually tracks, and (2) a screwed up portage > database (not the world file, but /var/db/portage/*). Note the > POTENTIALLY in there, it could be that your understanding of world is > correct and I'm misreading you, and/or it could be that the portage > database is fine and you are misreading events, and nothing is wrong. > > First, the world file. Note that the world file does *NOT* track > EVERYTHING that's merged, only the stuff directly merged from the command > line. Packages merged as dependencies of packages on the command line > aren't entered in the world file. The idea is that your world file tracks > the actual packages you want, not their dependencies, which may change > over time.
Yea, that was my understanding too but that's not right with portage <= 2.0.51.19. Since I had never read otherwise I was in the habit of invoking emerge thusly; "emerge -avDut --newuse pkg-name" and I was getting screwed by the --update. When that is used on an initial install of a package the package does not become listed in the world file. To correct that situation you can compare your current world file (back it up first) to the output of /usr/lib/portage/bin/regenworld. Jason Stubbs helped me sort through this last night in #gentoo-portage and pointed me to http://bugs.gentoo.org/show_bug.cgi?id=48181 for more details. That behavior is corrected so that anything that is entered explicitly on the command line is now added to the world file regardless of the use of the --update flag in upcoming versions of portage. The --deep option doesn't correct this errant behavior either. > This has a number of implications. First, it's what the --deep (-D) > option is all about. An existing package will work just fine with the > version of a dependency originally merged to support it, and won't tend to > use any new features in newer versions of that dependency anyway. Thus, > with just the --update (-u) flag, emerge will only update the stuff > you actually want -- the stuff in world (and your system profile, which > for the purposes of this discussion, can be considered dependencies of a > system virtual that's always presumed to be in world), naturally updating > any dependencies necessary to support new versions of the packages in > world, but /not/ updating dependencies where your currently merged version > is just fine, even if there are new versions of that dependency available. > Many folks, particularly those with slower systems, will prefer this, > since it means fewer packages have to be merged -- there may be several > updates to a dependency that aren't merged because they aren't needed for > stuff in your world file, saving that emerge time, between versions > actually NEEDED to support new versions of stuff you DO want directly -- > that IS in your world file. (Note that this won't cause security issues > with old versions either, because in that case they'll be masked, so > emerge will unmerge them at the update, and because the dependency is > still there, portage will see it needs something to fill it and will merge > the latest unmasked version, which should be a version containing the > security update. Thus, not updating dependencies under normal > circumstances during an update is fine, security-wise.) The --deep option > causes emerge to scan dependencies (as well as other packages in the > portage database but not in world -- which may be compile-time but not > runtime dependencies or may be no longer needed at all -- the ones emerge > depclean, see the next paragraph, would unmerge) and update them as well, > for those that want a fully updated system, even if it means more emerging > than necessary. > > The second implication triggers when dependencies change. If package > foo-1.x that's in your world file depends on package libbar, but you > upgrade to package foo-2.x, which no longer depends on libbar, but instead > on libbaz, libbaz will of course be merged, but you'll still have libbar > around. If nothing else depends on libbar, running emerge depclean will > then unmerge it, and any other packages no longer needed, **PROVIDED** > they aren't listed in your world file. If all dependencies are listed in > your world file, they'll all continue to remain around even when the > package that originally required them is long gone. Obviously, this isn't > something you want happening, even if you aren't worried about space, > because extra merged packages mean more stuff to wade thru when you are > trying to find something you WANT, AND more exposure to security issues > based on stuff that's hanging around that you don't even NEED. > > That takes care of the world file, but doesn't answer the now-logical > question of where sudo comes in, in that case. First, as stated above, > once merged and in the portage database, the --deep switch along with > --update will scan for updates of the package anyway. Still, why is it > there -- sudu isn't the kind of package that's likely to be a dependency? > To check for sure whether it's a dependency of what's in world or not, run > emerge depclean --pretend, and see if it's listed for unmerge. If so, you > know it's not depended on by anything in world and /should/ be safe to > unmerge -- provided your world file contains everything you /do/ want > specifically merged (it might not, and in fact might be a bit of a mess > at this point, if you've been merging stuff without understanding how the > world file works). If you want to unmerge it, you can, otherwise, add it > to your world file (echo app-admin/sudo >> /var/lib/portage/world. If it's > not listed there and isn't in your world file, it's gotta be a dependency > for something. You should be able to find out exactly /what/ by running an > emerge --pretend --tree --emptytree world and following the tree. As to > how it got merged in the first place if it's not a dependency and never > was, perhaps you emerged it at some point with the --oneshot option, which > will NOT put the stuff on the command line in the world file? Or, maybe > it was (is?) part of your system profile (which as I mentioned above makes > them implicit dependencies)? > > That should cover that aspect of your question, but there's still a loose > end to tie up -- regenworld. regenworld isn't always accurate, and may > add stuff to your world file that shouldn't be there, or (if your > emerge.log is incomplete) miss stuff that /should/ be there. Just because > the world file it creates doesn't match the one you had before, doesn't > mean the one you had was wrong. That's why you are supposed to make a > backup of the world file before running it. The tool is designed to be > used in emergencies, when you've deleted or corrupted your world file, to > create a replacement that can be used as a starting point. However, it's > not recommended that you rely on it, as it may be wrong. > > The other aspect of your question, and the reason I mentioned a possibly > corrupt portage database, is the twice in a row update of sudo. First, > it's possible the package /has/ been updated that fast. I know the one > update was for a security issue. I haven't run today's sync and update > check myself so I don't know if the package has been updated again or not, > but it's possible that the first update screwed up and they released > another one. A simply comparison of the version your emerge log says you > emerged yesterday, with what portage says you have currently and what it > wants to merge today, should reveal whether another update /has/ indeed > happened, or if it's trying to remerge the exact same version it did > yesterday. If it's still saying you have the old version merged, and > it's wanting to remerge the same one you did yesterday, your database may > be corrupted in some way. If it's a different version, then your db is > fine. > > I can't really give you any help on a corrupt db, but I /can/ say that at > one point, due to bad memory causing hard lockups at the time, I > apparently crashed in the middle of an update at some point, and for > several days, portage kept trying to emerge clean an old version of a > package that had been updated for several days, saying it did so > successfully each time, but then cleaning up the /same/ entry again, the > next time it did its emerge clean cycle. I never had to remerge the > package, portage had the emerge recorded successfully. It just somehow > failed to get the unmerge of the old package recorded correctly. I guess > it eventually took, because the problem went away on its own, after a week > or so. That "going away on its own" stuffs scares me and reminds me of Microshaft too much!?! I like to know what is happening with my system and I can see that you do too. Anyway, as stated above, when it is only --updated (even on initial install) it never gets added to the world file. > > Another question, at what point is a reboot required? I know that > > emerging a new kernel requires a reboot (at least to use it) but what > > about some of the other packages? Take glibc for instance, almost > > everything links to some part of it (except statically linked programs) > > so if you update that should you restart? It seems that the file > > locking and linking in the kernel should keep the old stuff in memory > > pages as long as it is being referenced by something but everything new > > should just use the new libraries. > > Linux is designed so that a reboot is normally not entirely necessary > unless you update your kernel and want to run the new one. When system > daemons you may be running are updated, the ebuilds generally trigger a > restart of the individual daemon, so that isn't normally necessary to do > manually. > > > Problems are going to arise with things like static variables (ones > > that have only one copy in memory regardless of the number of times > > called). Is there one copy of the variable for the library, one copy of > > the variable for each program that links to the library, one copy of the > > variable for each instance of each program, or one copy for each user? > > > > What about dynamically allocated internal data structures? Are they > > shared (w/ shared memory) between the two different library versions? > > Does each library (old & new) update their own copy blissfully unaware > > of the existence of the other? Many libraries exist on a Gentoo (and > > other GNU/Linux) system, I only picked glibc here because it is so high > > profile. > > System libraries, particularly glibc, must always exist. The way this is > usually accomplished is by creating the new library, then, in a single > operation, pointing the symlink at the new version that previously pointed > to the old version (using a command such as ln -sf /lib64/libc-2.3.5.so > /lib64/libc.so.6). After that, the old version may be removed. To ensure > that the new library is being used, you'll then want to restart anything > running that links to it, but newly launched applications will > automatically load the new version. Current processes that can > continue using the previous version of the library (or that don't load > it) need not be restarted. Note that just because an app is running does > NOT mean it has loaded everything it might need from linked libraries. > Sometimes, they are loaded only when called. There is however a switch to > the linker available that forces full linking at program launch. > This switch is traditionally used for SUID/SGID applications, due to > (generally theoretical) security issues with lazy linking and SUID/SGID. > Libraries are only shared in as much as they are identical, so if a newly > launched app finds an updated version to link against, that memory will > not be shared with the older copy in memory. I believe that a very nice solution for this was presented by Caitlin Ross in the other response. > I happen to use that switch here, after reading about it in connection > with a portage QA warning (now toned down, again, because it's > traditional/theoretical, nothing seriously urgent) having to do with that > SUID/SGID thing. > > LDFLAGS="-Wl,-z,now" > > The flag does mean apps take a bit longer to load, and with that load, > pull functions and libs into memory that I may not actually ever use > during this runtime session. However, there are positives as well. In > addition to the security thing, it's nice getting any errors about bad > links that might otherwise happen when it tried to load that function, up > front. Additionally, while initial load time might be a bit longer, it > means less wait when I initiate the action that would actually do the > load, later. Finally, with a gig of memory, most of which is normally > cache anyway, save for the initial launch of the first program that links > a library, it's nearly always all or mostly in cache memory anyway, so the > slowdown isn't really noticeable in the vast majority of cases. (One > exception would seem to be the initial load of KDE. I noticed HUGE > improvements in initial load time there, with gcc-4, likely only so > pronounced because of all the libraries I was loading then, that normally > would wait and be loaded later when actually accessed. Part of this was > due to the -fvisibility changes, no largely revoked pending revision and > integration into qt first, but it's still faster, it seems to me, than > kde-3.3 as compiled with gcc-3.4 was.) Yea, some of this in KDE has to do with "post binding" or where the code doesn't exactly know which function to link to until the code is running. This is actually a feature called inheritance and can be quite useful in languages such as C++, Objective C, Smalltalk, and many others. Briefly if goes like this: Imagine a data structure, employee, and three sibling data structures, hourly, salary, and executive. Now define a print function for all for of the data structures or classes listed above that prints each element of the class not forgetting to make employee->print() virtual so it is never called (you can't have an employee that's not either hourly, salary, or executive). Now define a function that takes in a list of employees and calls print for each one. That function is not going to know which of the three inherited print functions to call until it is running so it just calls employee->print() and that method (function defined within a class) will call the needed method of its children based upon what type of instance of data it was called with. Consider the following declaration and initialization (aka constructor): employee *data = new executive(); data is defined as a pointer to employee but created as a pointer to executive. Anyway, KDE does this sort of thing extensively, it is called post binding, and it slows down execution and load times a bit but is very useful. Think of a parent graphic object with children of types circle, line, rectangle (and square would inherit from rectangle), point, etc.. The save-as dialog could inherit from the load-file dialog since save-as has everything that load-file does but adds a text entry box for you to type in. > For rebuilds of X or GNOME/KDE/XFCE/whatever, you'll probably want to exit > your desktop environment and X, and restart them, but you shouldn't have > to reboot, or even restart any of the system daemons and the like. Yea, but I usually wait until it starts behaving like a typical M$ product. :-) > Remember, however, that with shells, the environment matters as much or > more than the version of shell running. Keep in mind the differences > between exported and unexported shell variables, and login vs regular > shells, in terms of what variables are inherited, vs the ones that are > not, vs the ones that get reset by the initialization process. With bash, > this can be /quite/ complex, because sub-shell instances use the same > executable, but get only the exported environment an external program > would get, while sourced instances are part of the same instance, executed > in place, while executing bash by name uses one set of rules (and files) > for initializing, while executing it using the sh alias uses an older more > POSIX-like set, and then there's login shell rules vs ordinary shell > rules, in terms of the files parsed (.profile, .bash_profile, .bashrc, and > the system versions of same, etc). As I said, /quite/ complex. It > /usually/ does what one would expect, but when it's not, figuring out just > where it's getting the values it gets can sometimes be an "interesting" > exercise. <g> At times, in addition to referring to a list of which > files get run at initialization under what conditions, I've had to resort > to sticking debug echo statements in various locations so I can see what's > being run in what order, and/or what value a variable actually has at a > specific point. Yes, unfortunately I've had to play this game before and I'm not real fond of it. I think I have that mostly sorted not though. > As should be fairly apparent by now, there's few upgrades that really > force a reboot, except to load a new kernel (and with the introduction of > kexec, even that may be about to change, altho to what extent depends to a > large degree on your definition of "reboot"). That said, for the desktop > or workstation user that isn't running any multi-week-long processes, it > can at times be more convenient to reboot than to restart a list of > individual processes manually. >From the LKML it would seem that the kexec function's first real (and safe) use is to handle a kernel oops. The production kernel loads and reserves a space for and loads a small data gathering kernel. In the event of an oops the production kernel is in a questionable state so I/O is immediately disabled (lest ye smoke thy data) and control is transfered to the mini-kernel which can snapshot the running kernel, memory, and swap and safely write that data out to storage. Then the mini-kernel can cold boot the system resetting all the hardware and everything to a known good state. Most of the kernel devs are against any other use of kexec since I/O with a questionable current can put your job and sanity at risk and bypassing the hardware reset brings in all the issues of sw-susp leaving hardware in unknown (or not completely known) states. > One reboot alternative is to switch init levels, down to init 1 to stop > nearly everything, then back to init 3 (or whatever) to restart most of > the system. Note that on Gentoo, unlike on many distributions and *ix > versions, init 1 does NOT mean single user mode that shuts down all the > virtual terminals and what may be running from them. Thus, it's possible > to keep a long-running process alive either in another virtual terminal > (or in the background in your current terminal), or add them to a > couple custom init levels, one with your usual other system stuff in it as > well, one without most of it, then switch between those levels, so the > long-running process doesn't stop. If I'm going to go that far down then I'm going to reboot. For me, once I leave X there is no reason NOT to reboot. I know servers are totally different beast though. > > I was reading the minutes of the gcc summit and it seems that the > > register allocator is going to get reworked again. > > I saw some of the same news, but that's /waayyy/ beyond me. I very > quickly realized I was hopelessly lost and hadn't the foggiest idea what > they were talking about, or what the effect on the real world was going to > be, even tho the individual words continued to make sense. <g> However, > I'm sure there'll be many writeups on it long before any such changes > affect me, so I'm not all that worried. I understand what they were talking about but not the nuance of it all. But I agree that the devs will be sure to let us know before it gets down to the user level. Regards, -- Tres -- [email protected] mailing list
