Re: [gentoo-portage-dev] Masked by corruption
On 02.01.2007, at 06:56, Zac Medico wrote: In =portage-2.1.2_rc4-r2 t does that now for installed package (see bug #158931). For /var/cache/edb/dep the sqlite module is available (requires pysqlite or python-2.5 with sqlite support enabled). Where can i find documentation about this? I use metadata-transfer at the moment, but all info i got was from the examples-section in man emerge and from the forum. Is there some official complete list of theese modules with some description? Thanks, Philipp -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] Masked by corruption
On Fri, Jan 12, 2007 at 12:18:34AM +0200, Philipp Riegger wrote: On 02.01.2007, at 06:56, Zac Medico wrote: In =portage-2.1.2_rc4-r2 t does that now for installed package (see bug #158931). For /var/cache/edb/dep the sqlite module is available (requires pysqlite or python-2.5 with sqlite support enabled). Where can i find documentation about this? I use metadata-transfer at the moment, but all info i got was from the examples-section in man emerge and from the forum. Is there some official complete list of theese modules with some description? The portage can probably answer better than I can, but here's a quick hack. grep 'class database' -rl /usr/lib/portage/pym/cache/ -- Robin Hugh Johnson Gentoo Linux Developer E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgplwrZfNf3z7.pgp Description: PGP signature
Re: [gentoo-portage-dev] Masked by corruption
On 1/12/07, Philipp Riegger [EMAIL PROTECTED] wrote: Where can i find documentation about this? I use metadata-transfer at the moment, but all info i got was from the examples-section in man emerge and from the forum. Is there some official complete list of theese modules with some description? http://gentoo-wiki.com/TIP_speed_up_portage_with_sqlite With a ruby script written by myself which does quick reverse dependancy lookups using that SQLite. ( portage still doesnt seem to utilize the SQLite database for that capacity :/ ) -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] Masked by corruption
Thanks for the replies; my apologies for the delay in repying back Based on the replies, I was able to track the problem down. The cdb patch is incompatable with the latest version of portage. Removing that from /etc/portage/modules allows portage to work. That said, there is still the bug of using the filesystem as a database. It now takes about thirty minutes for any emerge run to finish reading in /var/cache/edb/dep and most of /var/db/pkg. It is imperative that portage use a single file for those databases. (One per db is fine, in case the above is ambiguous.) This is the same problem that git ran into, leading to pack files, or that cnews and inn hit, leading to the rotating spool files. It may work reasonably well on a headless server, with only a few packages installed. But on a general purpose workstation with many packages installed it falls over hard. Incidently, the above 30 minute minimum run time is even when running with --nodeps. I'll also be posting a bug report about unmerging. Portage is stat(2)ing *every* file under a directory that might need removal rather than just checking whether there are any entries in the dir. Running stat(2) on everything in /usr/lib, /usr/bin, /usr/share/doc and similar completely thrashes the dirent cache. It is even worse than running ldconfig when no changes were made in dirs that ldconfig monitors. Thanks for the work; I hope to see performance improvements this year. -JimC -- James Cloos [EMAIL PROTECTED] OpenPGP: 1024D/ED7DAEA6 -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] Masked by corruption
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 James Cloos wrote: Thanks for the replies; my apologies for the delay in repying back Based on the replies, I was able to track the problem down. The cdb patch is incompatable with the latest version of portage. Removing that from /etc/portage/modules allows portage to work. That said, there is still the bug of using the filesystem as a database. It now takes about thirty minutes for any emerge run to finish reading in /var/cache/edb/dep and most of /var/db/pkg. It is imperative that portage use a single file for those databases. (One per db is fine, in case the above is ambiguous.) In =portage-2.1.2_rc4-r2 t does that now for installed package (see bug #158931). For /var/cache/edb/dep the sqlite module is available (requires pysqlite or python-2.5 with sqlite support enabled). This is the same problem that git ran into, leading to pack files, or that cnews and inn hit, leading to the rotating spool files. It may work reasonably well on a headless server, with only a few packages installed. But on a general purpose workstation with many packages installed it falls over hard. Incidently, the above 30 minute minimum run time is even when running with --nodeps. That's been fixed for bug #158649. If you make extensive use of PORTDIR_OVERLAY then you will probably want to run emerge --regen after each sync to avoid having to generate cache at other times. I'll also be posting a bug report about unmerging. Portage is stat(2)ing *every* file under a directory that might need removal rather than just checking whether there are any entries in the dir. Running stat(2) on everything in /usr/lib, /usr/bin, /usr/share/doc and similar completely thrashes the dirent cache. It is even worse than running ldconfig when no changes were made in dirs that ldconfig monitors. Thanks. Thats fixed in svn r5443. Thanks for the work; I hope to see performance improvements this year. -JimC Zac -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFmeX6/ejvha5XGaMRAugNAJ9fD272i73k7I+OPdLoKwT4L5LutQCbBdbB zV1QY1Wd2icP8AzagGcyg5E= =XLFd -END PGP SIGNATURE- -- gentoo-portage-dev@gentoo.org mailing list
[gentoo-portage-dev] Masked by corruption
I'm getting an inordinate number of masked- by corruption failures since merging portage-2.1.2_rc3-r7. Part of the problem is the addition of the complete dep graph. (Which makes testing extraordinarily and horrifically painful; it now takes emerge(1) about 20 to 30 minutes to get started, since reading in everything in /var/db/pkg thrashes the dcache) It is unable to deal with some of the stuff already installed. The corruption is not changed files. Hashes of the ebuilds always match the srings in the Manifest files. It seems that it just doesn't like old syntax. Running with --debug did not explain what it found to be corrupt. I don't see any relevant bugs, and I think this requires some discussion, hense this post. It'll take a couple of hours for another sync and update to run, should I even bother with -r8? -JimC -- James Cloos [EMAIL PROTECTED] OpenPGP: 1024D/ED7DAEA6 -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] Masked by corruption
Jim, I opened bug 158632, so that's worth looking at. It also points to a post on the forums about the problem. I can't add much to the discussion myself, just that it affected two of my machines differently which I found rather odd (one has difficulty with all versions of mpc, the other with only one version). I hope that helps... Mike 5:) -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] Masked by corruption
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 James Cloos wrote: I'm getting an inordinate number of masked- by corruption failures since merging portage-2.1.2_rc3-r7. Do you get a !!! Digest verification failed: message before that, or only the masked by: corruption message? Assuming that the messages are invalid, you may be able to use FEATURES=-strict or FEATURES=digest to avoid those messages. However, you should know exactly what those features do before you change them. Part of the problem is the addition of the complete dep graph. (Which makes testing extraordinarily and horrifically painful; it now takes emerge(1) about 20 to 30 minutes to get started, since reading in everything in /var/db/pkg thrashes the dcache) It is unable to deal with some of the stuff already installed. It sounds like you're experiencing this bug: https://bugs.gentoo.org/show_bug.cgi?id=158649 I'll get that one fixed ASAP. The corruption is not changed files. Hashes of the ebuilds always match the srings in the Manifest files. It seems that it just doesn't like old syntax. Running with --debug did not explain what it found to be corrupt. I don't see any relevant bugs, and I think this requires some discussion, hense this post. Please file a bug. It'll take a couple of hours for another sync and update to run, should I even bother with -r8? -JimC Don't bother because -r8 will behave the same way. Zac -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFiWaM/ejvha5XGaMRAgUgAJ41lhhIOaf9skhgl7PbKGdi0IlwOwCgv9+P rAG0JXBvX7m+25SPOZ5psr8= =nGqx -END PGP SIGNATURE- -- gentoo-portage-dev@gentoo.org mailing list
