Re: [gentoo-dev] Multiple Repo Support
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ciaran McCreesh wrote: > On Fri, 16 Dec 2005 00:36:54 -0500 Alec Warner <[EMAIL PROTECTED]> > wrote: > | emerge blar --repo=ciaranmssekritrepo > | > | This accomplishes the same thing, except I get to name the repo > | whatever I wish, and you lose the ability to specify repositories in > | DEPEND. > > ...and it stops you from being able to package.mask things in a given > repository, Ideally there will be per-repo package.mask entries as well, since .mask files are typically repository-based. > and it stops you from being able to stick to a given > repository for world entries, and it stops you from being able to use /var/lib/portage/world sucks on the whole. > it in all those zillion other locations where you can currently use a > dep atom to do something useful. Anything in particular other than "those other useful things"? > > Really, emerge blar::ciaranmssekritrepo with Portage creating the > 'restricted' world entry (and the same behaviour for SLOT deps) would > be so much more useful... > -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQ6JXY2zglR5RwbyYAQLnNA//W12tpNkaXMZ3y3qVKvOX2SsP/5gem/vY wZHeUn/h0zYFJ4cMvo/+at2Q6/+c4lIqpac8ihoxJowJX4UnXECk29vHRsZVOcMm GzO6aiB0MHd+IdBV/yLc3g8gLP2qOaFPIhnGzZw5wG87+nh6x4E9enZ50akgViSN gpYNF6fPhvjio7ugTYiXSyf3ScVWq/LP/Cf8TI4Dj9QeoeO/Lfg3sEq93JysctZs XxpiazkHk4X8p0Gpk62PCwEtk/uA/VYC8+4JysMi1t3UZyqcghREVNRzNu7w8ErU eonwty/PANReP9f74i7l6cV9y/HTrkkmmh/hxWhKqnuXAkki208D58f5vrkgTr/h nk41lWAQ0neh2sBQ4EPs8X7SjVgrxYnsZRicWDa7LosZpSOQwb47XZYZyHvTS0/z LyMaHC5JniGHis8XGOeI9nLnTRsiItoF+6DK15t8Pel2RyE2FsgkZWCwU0w4s8QE yC8d9ZW2/nim205jqDlpIZZJIQw7d1RAA+FTJNxDrlcSbxK/AjTajOajdY7wRU22 rmGMrkhXyXnIB6lmiwPYzYHoUi0SoJ8NwurjghjhePNGDfPbyVzKJmlaz9Hz3mo3 Z796P05FPW+EwiNv9MYAfXz5ImgSAmWekUcdiCl5VmWnqfNq+GW6q4fDB1z+f205 fD5eCWHusmk= =XC8n -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Multiple Repo Support
On Fri, 16 Dec 2005 00:36:54 -0500 Alec Warner <[EMAIL PROTECTED]> wrote: | emerge blar --repo=ciaranmssekritrepo | | This accomplishes the same thing, except I get to name the repo | whatever I wish, and you lose the ability to specify repositories in | DEPEND. ...and it stops you from being able to package.mask things in a given repository, and it stops you from being able to stick to a given repository for world entries, and it stops you from being able to use it in all those zillion other locations where you can currently use a dep atom to do something useful. Really, emerge blar::ciaranmssekritrepo with Portage creating the 'restricted' world entry (and the same behaviour for SLOT deps) would be so much more useful... -- Ciaran McCreesh : Gentoo Developer (I can kill you with my brain) Mail: ciaranm at gentoo.org Web : http://dev.gentoo.org/~ciaranm signature.asc Description: PGP signature
Re: [gentoo-dev] Multiple Repo Support
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ciaran McCreesh wrote: > On Thu, 15 Dec 2005 22:34:05 -0500 Andrew Muraco <[EMAIL PROTECTED]> > wrote: > | 2. What choices/options are on the table for this feature? > > The big controversy seems to be over whether repositories carry a > unique identifier string (for example, in metadata/repository_id) or > whether it's user-assigned. The former is clearly the more sensible > option, since it lets you do things like (syntax made up): > > DEPEND=">=foo-bar/baz-2.1::ciaranmssekritrepo" > Well lets return to normal syntax for a moment. DEPEND=">=foo-bar/baz-2.1" And lets assume this package is named "blar" because I enjoy that name. emerge blar --repo=ciaranmssekritrepo This accomplishes the same thing, except I get to name the repo whatever I wish, and you lose the ability to specify repositories in DEPEND. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQ6JSdWzglR5RwbyYAQJsbg/7B8XgQlR6bcTOyeLG2IrguyDh9HJJfhlP HtiypusFdXF35mFhS469Tsc/dIXPCFbVf7OnflO+m7MCiwryQu19v58R+K5dZgli lkObsiLmafsdXLE5TGlJ7CuB0yboHrjR/hT8n7tomRuQ/g4YcpvCCL96eSlaQ5iX tpebI/U0VzOHayoWeGXeMp4oEN197eSg4IM8Q5TQgwh84boDnj/gZAuY11g0nUCL l3Ardv1qjWHwhmVDnKSxF6fR6EAug0ldNNFL94+xRw1r9Z9pIchC4OO90SBRx6dl MuwvQLCo9N+HaZgztcXUR0uUFE2H7sTjTVHiIW4KUfZYvoslvNRq9SLBCkn39fQp LSO4cIpKq81Tov7Ngk/bx7NYfcwv34X6q0ezKCfE4ZYdilST0189Jd6NN2/SiGy6 HOdQh1YWre2jQbcS54Z7p+DSOX6XBg3yUQZTtxDKlaP4vTJdMVjUgqSXdLGFCnGV suDshDnNQY4opFzmu158U36vX10H5DLqLTTlrJ+3igzb9msnQ/CVnT+lbUFACpq2 DzrBOuJBJcCq+5KPoBk295VozOILjK2hGo+uLLqhA43G4K+mxOD1ER1SOo4osfF2 YBU26fhm14Xxzcf64MtOKB5EzaewX12uzBPFh/a3Ps9VEWCOadhELO0xSENm+ewP CwxPxIcmsTw= =HKbW -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Multiple Repo Support
Curtis Napier wrote:
Ciaran McCreesh wrote:
On Thu, 15 Dec 2005 22:34:05 -0500 Andrew Muraco <[EMAIL PROTECTED]>
wrote:
| 2. What choices/options are on the table for this feature?
The big controversy seems to be over whether repositories carry a
unique identifier string (for example, in metadata/repository_id) or
whether it's user-assigned. The former is clearly the more sensible
option, since it lets you do things like (syntax made up):
DEPEND=">=foo-bar/baz-2.1::ciaranmssekritrepo"
which would add a restriction that only packages in ciaranmssekritrepo
would be considered. This only works if the repository knows its own
identifier, however...
Incidentally, the ::repo syntax (or whatever) would also be useful in
the world file, along with :slot. So something like:
foo-bar/baz:2::ciaranmssekritrepo
would tell the package manager that you want baz SLOT 2 from
ciaranmssekritrepo.
*shrug* But it seems the Portage guys want repository names to be
user-assigned, which makes them far less useful.
This functionality would come in very very handy. Would user assigned
repository names be able to mimic this functionality somehow?
What about giving each repo an identifier? That way repos with ebuilds
can have the repo_id in the ebuilds and not have to worry, repo_ids
would take precedence when in conflict. Identifiers would just be
provided for user-stuff, like ex # emerge sync MyRepo or # emerge -av
MyRepo://foo ? [foo being the name of a package in repo tree with the
identifier of MyRepo]
what about having a portage config file
/etc/portage/repositories:
priority MyRepo,gentoo
repository {
identifer = "gentoo"
rsync= "rsync://../"
repo_path= "/usr/portage"
}
repository {
identifer = "MyRepo"
rsync= "rsync://../"
repo_path= "/usr/MyRepoTree"
}
(an example for someone that wants to have MyRepo be prefered over
gentoo tree)
Just tossing out ideas,
Andrew Muraco
--
gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Multiple Repo Support
Ciaran McCreesh wrote: On Thu, 15 Dec 2005 22:34:05 -0500 Andrew Muraco <[EMAIL PROTECTED]> wrote: | 2. What choices/options are on the table for this feature? The big controversy seems to be over whether repositories carry a unique identifier string (for example, in metadata/repository_id) or whether it's user-assigned. The former is clearly the more sensible option, since it lets you do things like (syntax made up): DEPEND=">=foo-bar/baz-2.1::ciaranmssekritrepo" which would add a restriction that only packages in ciaranmssekritrepo would be considered. This only works if the repository knows its own identifier, however... Incidentally, the ::repo syntax (or whatever) would also be useful in the world file, along with :slot. So something like: foo-bar/baz:2::ciaranmssekritrepo would tell the package manager that you want baz SLOT 2 from ciaranmssekritrepo. *shrug* But it seems the Portage guys want repository names to be user-assigned, which makes them far less useful. This functionality would come in very very handy. Would user assigned repository names be able to mimic this functionality somehow? -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Multiple Repo Support
On Thu, 15 Dec 2005 22:34:05 -0500 Andrew Muraco <[EMAIL PROTECTED]> wrote: | 2. What choices/options are on the table for this feature? The big controversy seems to be over whether repositories carry a unique identifier string (for example, in metadata/repository_id) or whether it's user-assigned. The former is clearly the more sensible option, since it lets you do things like (syntax made up): DEPEND=">=foo-bar/baz-2.1::ciaranmssekritrepo" which would add a restriction that only packages in ciaranmssekritrepo would be considered. This only works if the repository knows its own identifier, however... Incidentally, the ::repo syntax (or whatever) would also be useful in the world file, along with :slot. So something like: foo-bar/baz:2::ciaranmssekritrepo would tell the package manager that you want baz SLOT 2 from ciaranmssekritrepo. *shrug* But it seems the Portage guys want repository names to be user-assigned, which makes them far less useful. -- Ciaran McCreesh : Gentoo Developer (I can kill you with my brain) Mail: ciaranm at gentoo.org Web : http://dev.gentoo.org/~ciaranm signature.asc Description: PGP signature
[gentoo-dev] December 15th Meeting Summary
this months meeting wasnt too eventful, kind of quiet ... on the agenda: - Marius: decision on multi-hash for Manifest1 there was a bit of hearsay about why the council was asked to review/decide on this issue since we werent able to locate any portage devs at the time of the meeting ... so our decision comes with a slight caveat. assuming the reasons our input was asked for was summarized in the e-mail originally sent by Marius [1], then we're for what we dubbed option (2.5.1). that is, the portage team should go ahead with portage 2.0.54 and include support for SHA256/RMD160 hashes on top of MD5 hashes. SHA1 should not be included as having both SHA256/SHA1 is pointless. further more, we hope this is just a hold over until Manifest2 is ironed out/approved/implemented/deployed. it was also noted that we should probably omit ChangeLog and metadata.xml files from the current Manifest schema as digesting them serves no real purpose. [1] http://article.gmane.org/gmane.linux.gentoo.devel/33434 - Council: portage signing shortly after our November meeting, a nice summary was posted by Robin Johnson that covered signing issues from top to bottom. as such, it was felt that trying to throw together a GLEP would not be beneficial. instead we will be adding a constant agenda item to future council meetings as to the status of portage signing issues to keep the project from slipping into obscurity again. full meeting log: http://www.gentoo.org/proj/en/council/meeting-logs/20051215.txt -mike -- gentoo-dev@gentoo.org mailing list
[gentoo-dev] Multiple Repo Support
As i sit reading the current list of list emails about GLEP 42 I see that the topic of Multiple Repos coming up over and over again. I wanted to ask to see where that support is, and based on what feedback help move along so that a standard can be produced. So, now with a few short questions: 1. Would Multiple Repo support be GLEP worthy? 1.1. If so, Should I write a GLEP, or poke at some dev to do it? (i'm more then willing and able to write a GLEP if that is what is required.) 2. What choices/options are on the table for this feature? 2.1. Which routes are more likely to be implemented? 2.2. Which method would you like to see? 2.3. Does backwards compatiblity really matter, as long as it doesn't break people that are using older portage? (ex. once portage is upgraded it will move x to y/x and so older versions wouldn't work, but since only the new version would be installed it wouldn't be an issue.) Now, that I've asked for some information I want to share one (hackish, I guess) way that it could be done with minimal changing. in /etc/make.conf: RSYNC_REPONAME="rsync://.../" with RSYNC=" .." being defaultly called 'gentoo' or 'portage' that REPONAME would be used for the tree's folder name /usr/repositories/REPONAME/ and 'emerge sync REPONAME' would sync only that repo, or 'emerge sync [all]' would sync all Anyways, thats just a quick thought I had on the topic. Flames, comments, questions (and answers) welcome Andrew Muraco -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC: disallowing multiple votes per person in council meetings
On Thu, Dec 15, 2005 at 08:20:36PM +, Ciaran McCreesh wrote: > Possible proposal: the current council meeting rules be updated with > one of the following two clauses: > > > A proxy must not be an existing council member, and any single person > > may not be a proxy for more than one person at any given meeting. > > (The difference being, the former allows a council member to appoint > another council member as their proxy, so long as said member forfeits > their own role.) I think the latter makes more sense - only allowing non-council members to represent absent council members. That is how it works in the boards I have been involved with. > * It will lead to increased discussion, which in turn means the council > is more likely to notice any problems with items on the agenda. > > * It will avoid having council meetings made up of two or three council > members, all acting as proxies for other members. More eyes on the matter is a good thing. If we allow council members to proxy for other council members we might end up with quite a few eyes on the discussion... > * This fits in better with the way I was intending proxies to work when > I wrote the slacker boot proposal, rather than the way they've ended up > working due to insufficient pedantry in the original description :) Sounds good to me. > Arguable disadvantage: > > * It makes it harder for council members to all go "oops, can't make > it, so vapier is my proxy" at the last minute. Yeah, well - I think we can live with that. How many times has the proxy function been used so far? > On the same subject, I'd also like to see the "meeting participants" > table updated to explicitly list proxies, for example in the form > "jaervosz (for koon)". Yes, that would be natural. ./Brix -- Henrik Brix Andersen <[EMAIL PROTECTED]> Gentoo Metadistribution | Mobile computing herd pgpiFiWsgzNxs.pgp Description: PGP signature
[gentoo-dev] RFC: disallowing multiple votes per person in council meetings
Possible proposal: the current council meeting rules be updated with one of the following two clauses: > Each person at a council meeting may represent only one voting role. Or: > A proxy must not be an existing council member, and any single person > may not be a proxy for more than one person at any given meeting. (The difference being, the former allows a council member to appoint another council member as their proxy, so long as said member forfeits their own role.) Advantages: * It would allow voters to see how people are likely to behave if they are voted in in future elections. (Assumption: many people who are appointed as proxies will stand in future elections.) * It will lead to increased discussion, which in turn means the council is more likely to notice any problems with items on the agenda. * It will avoid having council meetings made up of two or three council members, all acting as proxies for other members. * This fits in better with the way I was intending proxies to work when I wrote the slacker boot proposal, rather than the way they've ended up working due to insufficient pedantry in the original description :) Arguable disadvantage: * It makes it harder for council members to all go "oops, can't make it, so vapier is my proxy" at the last minute. On the same subject, I'd also like to see the "meeting participants" table updated to explicitly list proxies, for example in the form "jaervosz (for koon)". Thoughts? -- Ciaran McCreesh : Gentoo Developer (I can kill you with my brain) Mail: ciaranm at gentoo.org Web : http://dev.gentoo.org/~ciaranm signature.asc Description: PGP signature
Re: [gentoo-dev] Optimizing performance
Wernfried Haas wrote: On Thu, Dec 15, 2005 at 09:13:34AM -0500, Chris Gianelloni wrote: There was a tip in the GWN about turning on dir_index on an already formatted file system. If formatting a new one, just use mkfs.ext2 -J -O dir_index /dev/$whatever to create your file system. Good thing you remind me of that. As a new ext3 convert (i happily used reiser3 for years before), any problems to be expected by doing so? Afaics it turns on B-trees which should have no impact on the safety of my data, right? Just want to make sure, i rather use a slightly slower file system than risking data loss. cheers, Wernfried I've been using it on ext3 for about a year and it has never lost any data. It also seems to speed up emerge --sync about as much as using reiser3 does. -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Optimizing performance
On Thursday 15 December 2005 04:48, Patrick Lauer wrote: > Hi all, > > I was wondering if there are any sane ways to optimize the performance > of a Gentoo system. > Overoptimization (the well known "-O9 -fomgomg" CFLAGS etc.) tends to > make things unstable, which is of course not what we want. The "easy" > way out would be buying faster hardware, but that is usually not an > option ;-) > > So ... what can be done to get the stable maximum out of your hardware? This should be obvious, but don't USE=debug globally. Last time I did that, it made my Athlon64 3400+ with 1G of RAM feel like the 300MHz PII with 192M of RAM I have. -- # # electronerd, the electronerdian from electronerdia # pgpCQn0k0Wn8f.pgp Description: PGP signature
Re: [gentoo-dev] Optimizing performance
On Thu, Dec 15, 2005 at 09:13:34AM -0500, Chris Gianelloni wrote: > CFQ is much worse for a desktop system. I tend to like deadline for > playing games. These can probably make a bit more difference than a new > -fomg-itsofast-and-broken-math added to CFLAGS. That's funny, i switched from default to CFQ on my notebook which has a rather slow disk and it feels much better, especially when recovering from suspend to disk which swaps out a lot. It's possible it decreases overall performance, but it may feel faster sometimes. > There was a tip in the GWN about > turning on dir_index on an already formatted file system. If formatting > a new one, just use mkfs.ext2 -J -O dir_index /dev/$whatever to create > your file system. Good thing you remind me of that. As a new ext3 convert (i happily used reiser3 for years before), any problems to be expected by doing so? Afaics it turns on B-trees which should have no impact on the safety of my data, right? Just want to make sure, i rather use a slightly slower file system than risking data loss. cheers, Wernfried -- Wernfried Haas (amne) - amne at gentoo dot org Gentoo Forums: http://forums.gentoo.org IRC: #gentoo-forums on freenode - email: forum-mods at gentoo dot org -- gentoo-dev@gentoo.org mailing list
[gentoo-dev] Re: Re: Optimizing performance
Diego 'Flameeyes' Pettenò posted <[EMAIL PROTECTED]>, excerpted below, on Thu, 15 Dec 2005 17:03:59 +0100: > On Thursday 15 December 2005 16:43, Patrick Lauer wrote: >> [talking about -Os if I'm right] >> I've seen some reproducable breakage, e.g. KDE doesn't like it at all > Actually, I'm running KDE with -Os right now... Same here, and I've been running it with -Os over a year, IIRC, even when it was supposedly causing issues. Of course, I'm on amd64, which might have something to do with it, if the issues were x86(32) only. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman in http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Optimizing performance
Patrick Lauer wrote: > -user has the risk of many "use teh -fomglol flag, it si teh fast0r" ;-) > hardened doesn't have much to do with performance (although I'd be > interested what impact - if any - the different security features have!) fresh of typing (but worked on for few months) http://www.pjvenda.org/linux/doc/pax-performance/ -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Optimizing performance
On 12/15/05, Patrick Lauer <[EMAIL PROTECTED]> wrote: > > > Are there any application-specific tweaks (e.g. "use the prefork MPM > > > with apache2")? [...] > > is'n there "ab" [1] for apache testing ? > Yes, but that's apache specific and is quite hard to use correctly. Isn't that what you asked? -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Optimizing performance
On Thursday 15 December 2005 16:43, Patrick Lauer wrote: > [talking about -Os if I'm right] > I've seen some reproducable breakage, e.g. KDE doesn't like it at all Actually, I'm running KDE with -Os right now... -- Diego "Flameeyes" Pettenò - http://dev.gentoo.org/~flameeyes/ Gentoo/ALT lead, Gentoo/FreeBSD, Video, AMD64, Sound, PAM, KDE pgpxYRkW4Ai7s.pgp Description: PGP signature
Re: [gentoo-dev] Optimizing performance
On Thu, 2005-12-15 at 14:43 +0100, Francesco Riosa wrote: > having more than one disk or a lot of memory add very interesting > addition, read raid 0 (stripe) or tmpfs for working data that does'nt > need a backup fex: $PORTIR, /var/tmp ... tmpfs has miserable performance when larger than RAM iirc - you'd need >5G for openoffice :-) > I've found that preemption with the new standard 250Hz of the kernel is > suitable for mostly needs, however no server here has preemption enabled ;-) My system still manages to run a DVD at a load of ~8, so from my point of view that is not a problem (2Ghz Athlon ... one of the "faster" machines I'd say as many people still use ~500Mhz) What causes more problems are packages that become slow on update - e.g. gtk+ 2.8 is _really_ slow (takes a few seconds to redraw apps that took <1sec with 2.6 ... :-( ) > what is a normal workload ? Define it and creating tests should not be > so difficult, then there are apps that can help to profiling, thinking > at bootchart, sysproof, memproof, valgrind ... strace I guess then you'd have to split into server / desktop / ... > reiserfs is sustainable, at least for 99.999% of uses, last reiserfs bug > on very high load (and with degraded raid5) is dated 4 years ago here. > However upstream is going to the route of reiser4, much more complex, > and much more unstable, latest problems where in 2.6.14, additionally no > devs in gentoo are (will?) support it the patch for grub it's still not > in place I think. reiser4 is "new and untested", I'd keep away from it until it has shown its reliability. Also in my (limited) testing it is relatively slow (about the same speed as reiser3) > > Are there any application-specific tweaks (e.g. "use the prefork MPM > > with apache2")? What is known to break things, what has usually > > beneficial behaviour? Are there any useful benchmarks that show the > > performance difference between different settings? > is'n there "ab" [1] for apache testing ? Yes, but that's apache specific and is quite hard to use correctly. (but very nice for slashdotting simulation ;-) ) Patrick -- Stand still, and let the rest of the universe move signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Optimizing performance
On Thu, 2005-12-15 at 13:48 +0100, Patrick Lauer wrote: > > I was wondering if there are any sane ways to optimize the performance > of a Gentoo system. for package in $system_packages; do profile_application $package eliminate_bottlenecks $package submit_patch_upstream $package done Plus, you can fix bugs while you're at it! ;) -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Optimizing performance
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Patrick Lauer wrote: | On Thu, 2005-12-15 at 07:43 -0700, Duncan wrote: |>This really belongs on user, or perhaps on the appropriate purposed list, |>desktop or hardened or whatever, not on devel. That said, some |>comments... (I can't resist. ) | | -user has the risk of many "use teh -fomglol flag, it si teh fast0r" ;-) | hardened doesn't have much to do with performance (although I'd be | interested what impact - if any - the different security features have!) ~From http://www.gentoo.org/main/en/lists.xml -- gentoo-performance Discussions about improving the performance of Gentoo Although it was a bit quiet last time I was subscribed to it. Thanks, Donnie -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDoZC3XVaO67S1rtsRAvIhAKDOq5aL7mSWi4sv1Qvmvn/woVLKMwCgp5xG ap+Fg5bDbSF9ZcvGnS7ysuY= =0vj5 -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Optimizing performance
On Thu, 2005-12-15 at 07:43 -0700, Duncan wrote: > > I was wondering if there are any sane ways to optimize the performance > > of a Gentoo system. > This really belongs on user, or perhaps on the appropriate purposed list, > desktop or hardened or whatever, not on devel. That said, some > comments... (I can't resist. ) -user has the risk of many "use teh -fomglol flag, it si teh fast0r" ;-) hardened doesn't have much to do with performance (although I'd be interested what impact - if any - the different security features have!) > > - don't overtweak CFLAGS. "-O2 -march=$your_cpu_family" seems to be on > > average the best, -O3 is often slower and can cause bugs > > A lot of folks don't realize the effect of cache memory on optimizations. > I'll be brief here, but particularly for things like the kernel that stay > in memory, -Os can at times work wonders, because it means more of the > working set stays in a cache closer to the CPU, and the additional speed > in retrieving that code far outweighs the compromises made to > optimizations to shrink it to size. Conversely, media streaming or > encoding apps are constantly throwing out old data and fetching new data, > and the optimizations are often more effective for them, so they work > better with -O2 or even -O3. I've not seen any substantial benefits from -Os over -O2. Also the size difference is quite small - ~5M on a "normal" install iirc > There have been occasional problems with -Os, generally because it isn't > used as much and gets less testing, so earlier in a gcc cycle series. > However, I run -Os here (amd64) by default, and haven't seen any issues > that went away if I reverted to -O2, over the couple years I've been > running Gentoo. I've seen some reproducable breakage, e.g. KDE doesn't like it at all > (Actually, that has been the case, even when I've edited > ebuilds to remove their stripflags calls and the like. Glibc and xorg > both stripflags including -Os. xorg seemed to benefit here from -Os after > I removed the stripflags call, while glibc worked but seemed slower. Note > that editing ebuilds means if it breaks, you get to keep the pieces!) ... which is exactly what I wanted to avoid. Ricing for the sake of it is boring ;-) > For gcc, -pipe doesn't improve program optimization, but will make > compiling faster. -fomit-frame-pointers makes smaller applications if > you aren't debugging. Those are both common enough to be fairly safe. agreed > -frename-registers and -fweb may also be useful. (-fweb ceases to be so on > gcc4, however, because it is implemented differently.) -funit-at-a-time > (new to gcc-3.4, so don't try it with gcc-3.3) may also be worth looking > into, altho it's already enabled by -Os. These latter flags are less > commonly used, however, thus less well tested, and may therefore cause > very occasional problems. (-funit-at-a-time was known to do so early in > the 3.4 cycle, but those issues should have been long ago dealt with by > now.) I consider those /reasonably/ conservative, and it's what I run. > If I were running a server, however, I'd probably only run -O2 and the > first two (-pipe and -fomit-frame-pointers). on a server you'd not use omit-frame-pointers to keep debuggability I think. > Do some research on -Os, in any case. It could be well worth your time. from my (limited) experience it isn't, especially on CPUs with larger caches > This suggestion does involve hardware, but not a real heavy cost, and the > performance boost may be worth it. That's usually not an option :-) > Consider running a RAID system. I > recently switched to RAID, a four-disk setup, raid1/mirrored for /boot, > raid6 (for redundancy) for most of the system, raid0/striped (for speed) > for /tmp, the portage dir, etc, stuff that was either temporary anyway, or > could easily be redownloaded. (Swap can also be striped, set equal > partitions on each disk and set equal priority for them in fstab.) I was > very pleasantly surprised at how much of a difference it made! Yes. 4-disk raid5 delivers amazing performance with minimal CPU overhead (~10% @1Ghz) But 4 disks at 100Euro + controller (100 Eur) is more than the price of a "new" system for most people. > If you have > onboard SATA and are buying new disks so can buy SATA anyway (my case), > that should do just fine, as SATA runs a dedicated channel to each > drive anyway. SCSI is a higher cost option, ruled out here, but SATA > works very nicely, certainly so for me. SCSI does deliver better performance, but at a prohibitive cost for "average" users. > Again, a reasonable new-hardware suggestion. When purchasing a new system > or considering an upgrade, more memory is often the most effective > optimization you can make (with the raid suggestion above very close to > it). "The only thing better than a large engine is a larger engine" ;-) Depending on workload 4G does wonders, but again - prohibitive for the normal user. > Slower CPU and more memory, u
Re: [gentoo-dev] Creating a Sketeton System
No, only the config files that are in a stage 3 should be left, some of those will be edited and some will have been upgraded so they should be left. It would be like emerge --unmerge --shallow world to take you back the that original state so then any major changes could be made without reinstalling as you would basically have an upgraded stage 3 after some unmergingOn 12/12/05, Donnie Berkholz <[EMAIL PROTECTED]> wrote: George Prowse wrote:> yes but if you extracted a stage on to an already built system you would> not only have the the mess there that you wanted to get rid of but also> all your config files would revert back to older versions and you'd lose > any changes made> because i'd rather not use depclean but also depclean doesn't get rid of> the configs left by any packages, for instance: if i had xfce on my> system before and i did emerge -C xorg-x11 && emerge depclean xfce > would be wiped off but if i emerged xfce again there would still be> modified parts that would use the options i selected on the previous> version.It really sounds like you're contradicting yourself here. You don't want your config files overwritten, but you don't want your config files usedwhen you remerge the packages?Thanks,Donnie--gentoo-dev@gentoo.org mailing list
[gentoo-dev] Re: Optimizing performance
Patrick Lauer posted <[EMAIL PROTECTED]>, excerpted below, on Thu, 15 Dec 2005 13:48:05 +0100: > I was wondering if there are any sane ways to optimize the performance > of a Gentoo system. This really belongs on user, or perhaps on the appropriate purposed list, desktop or hardened or whatever, not on devel. That said, some comments... (I can't resist. ) > Overoptimization (the well known "-O9 -fomgomg" CFLAGS etc.) tends to > make things unstable, which is of course not what we want. The "easy" > way out would be buying faster hardware, but that is usually not an > option ;-) > > So ... what can be done to get the stable maximum out of your hardware? > > In my experience (x86 centric - do other arches have different > "problems"?) the following is stable, but not necessarily the optimum: The general rules are the same, but there are architectural differences that often change the details. I /think/ it was MIPS that has extremely slow i/o (I saw that mentioned in the split-kde-ebuilds debate, they said it could cause compile times to double -- a big thing for something as big as KDE). x86 (32-bit) has a relatively small number of CPU registers, compared to most other archs (amd64 in 64-bit mode increased the number dramatically, tho it's the same for 32-bit mode for compatibility reasons), and this has a big effect on register use strategy. That said, in the general case, the -march switch normally chooses pretty good defaults for the target arch. Modifying them a whole lot from that, other than to cover special cases, or with the general -Ox optimization switches, is therefore often counterproductive and/or problematic. > - don't overtweak CFLAGS. "-O2 -march=$your_cpu_family" seems to be on > average the best, -O3 is often slower and can cause bugs A lot of folks don't realize the effect of cache memory on optimizations. I'll be brief here, but particularly for things like the kernel that stay in memory, -Os can at times work wonders, because it means more of the working set stays in a cache closer to the CPU, and the additional speed in retrieving that code far outweighs the compromises made to optimizations to shrink it to size. Conversely, media streaming or encoding apps are constantly throwing out old data and fetching new data, and the optimizations are often more effective for them, so they work better with -O2 or even -O3. There have been occasional problems with -Os, generally because it isn't used as much and gets less testing, so earlier in a gcc cycle series. However, I run -Os here (amd64) by default, and haven't seen any issues that went away if I reverted to -O2, over the couple years I've been running Gentoo. (Actually, that has been the case, even when I've edited ebuilds to remove their stripflags calls and the like. Glibc and xorg both stripflags including -Os. xorg seemed to benefit here from -Os after I removed the stripflags call, while glibc worked but seemed slower. Note that editing ebuilds means if it breaks, you get to keep the pieces!) For gcc, -pipe doesn't improve program optimization, but will make compiling faster. -fomit-frame-pointers makes smaller applications if you aren't debugging. Those are both common enough to be fairly safe. -frename-registers and -fweb may also be useful. (-fweb ceases to be so on gcc4, however, because it is implemented differently.) -funit-at-a-time (new to gcc-3.4, so don't try it with gcc-3.3) may also be worth looking into, altho it's already enabled by -Os. These latter flags are less commonly used, however, thus less well tested, and may therefore cause very occasional problems. (-funit-at-a-time was known to do so early in the 3.4 cycle, but those issues should have been long ago dealt with by now.) I consider those /reasonably/ conservative, and it's what I run. If I were running a server, however, I'd probably only run -O2 and the first two (-pipe and -fomit-frame-pointers). Do some research on -Os, in any case. It could be well worth your time. > - check that all IDE disks use DMA mode, otherwise they are limited to > ~16M/s with a huge CPU usage penalty. Sometimes (application-specific) > increasing the readahead with hdparm gives a huge throughput boost. This suggestion does involve hardware, but not a real heavy cost, and the performance boost may be worth it. Consider running a RAID system. I recently switched to RAID, a four-disk setup, raid1/mirrored for /boot, raid6 (for redundancy) for most of the system, raid0/striped (for speed) for /tmp, the portage dir, etc, stuff that was either temporary anyway, or could easily be redownloaded. (Swap can also be striped, set equal partitions on each disk and set equal priority for them in fstab.) I was very pleasantly surprised at how much of a difference it made! Cost, as I said, is reasonable, particularly if you have disks laying around or can buy them used. Even buying say three 80-gig drives and doing what I did only with a raid5 is reasonable,
Re: [gentoo-dev] Optimizing performance
On Thu, 2005-12-15 at 13:48 +0100, Patrick Lauer wrote: > - don't overtweak CFLAGS. "-O2 -march=$your_cpu_family" seems to be on > average the best, -O3 is often slower and can cause bugs -O2 -march=$your_cpu_family -pipe -fomit-frame-pointer -pipe Use pipes rather than temporary files for communication between the various stages of compilation. This fails to work on some systems where the assembler is unable to read from a pipe; but the GNU assembler has no trouble. -O also turns on -fomit-frame-pointer on machines where doing so does not interfere with debugging. (However, x86 is not one of these machines, so you can turn it on if you are not a developer doing debugging for a slight additional speed increase) -fomit-frame-pointer Don't keep the frame pointer in a register for functions that don't need one. This avoids the instructions to save, set up and restore frame pointers; it also makes an extra register available in many functions. > - don't do anything with ASFLAGS, LDFLAGS. This causes weird random > breakage (e.g. LDFLAGS="-O1" causes prelink to fail with "absurd" > errors) and doesn't give a noticeable performance boost Correct. Also, running prelink can improve speed at the cost of disk space. > - check that all IDE disks use DMA mode, otherwise they are limited to > ~16M/s with a huge CPU usage penalty. Sometimes (application-specific) > increasing the readahead with hdparm gives a huge throughput boost. I typically use the same hdparm settings as listed in the Handbook: disc0_args="-d1 -A1 -m16 -u1 -a64 -c1" cdrom0_args="-d1 -c1" > - kernel tweaks like preempt may increase the responsiveness of the > system, but often reduce throughput and may have unexpected sideeffects > like random audio stutter as well as random kernel crashes ;-) This is especially true on non-x86 architectures. > - kernel tweaks like setting swappiness or using a different I/O > scheduler (CFQ, deadline) should help, but I'm not aware of any "real" > benchmarks except microbenchmarks (can create 1M files 10% faster! - > yes, but how does it behave with a normal workload?) CFQ is much worse for a desktop system. I tend to like deadline for playing games. These can probably make a bit more difference than a new -fomg-itsofast-and-broken-math added to CFLAGS. > - using a "smarter" filesystem can dramatically improve performance at > the potential cost of reliability. As data on FS reliability is hard to > find from unbiased sources this becomes a religious issue ... migrating > from ext3 to reiserfs makes "emerge sync" extremely much faster, but is > reiserfs sustainable? Well, reiserfs 3 isn't so bad on architectures where it doesn't vomit all over itself immediately. Also, resierfs loses much of its luster if you're running ext3 with dir_index. There was a tip in the GWN about turning on dir_index on an already formatted file system. If formatting a new one, just use mkfs.ext2 -J -O dir_index /dev/$whatever to create your file system. > Are there any application-specific tweaks (e.g. "use the prefork MPM > with apache2")? What is known to break things, what has usually > beneficial behaviour? Are there any useful benchmarks that show the > performance difference between different settings? Well, turning on SBA and Fast Writes on Nvidia always helps. As for benchmarks, I think the issue is it depends entirely on usage. Having something that is 30% faster on paper isn't very useful if you never do it the way the benchmark does. I wish I had more numbers/examples here, but there isn't really much in the way of decent benchmarks published and readily available. Hopefully some other people will know of more of them than I do. -- Chris Gianelloni Release Engineering - Strategic Lead x86 Architecture Team Games - Developer Gentoo Linux signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Optimizing performance
On Thursday 15 December 2005 14:43, Francesco Riosa wrote: > Some upstreams, mostly media related but also unsuspectable like MySQL, > use and test their apps with high optimizations. Not exactly true.. many media related upstreams forces "ricing" flags (-fomg-so-fast) on packages, but that does not really mean it's more stable that way... actually, xine-lib proved to be way more stable while *not* using ricing CFLAGS. The actual problem there is that many packages have code that breaks if you remove those flags, so for example xine-lib has to foce a few flags on to work fine (on x86). -- Diego "Flameeyes" Pettenò - http://dev.gentoo.org/~flameeyes/ Gentoo/ALT lead, Gentoo/FreeBSD, Video, AMD64, Sound, PAM, KDE pgpKksvz07bOr.pgp Description: PGP signature
Re: [gentoo-dev] Optimizing performance
Patrick Lauer wrote: > Hi all, > > I was wondering if there are any sane ways to optimize the performance > of a Gentoo system. > Overoptimization (the well known "-O9 -fomgomg" CFLAGS etc.) tends to > make things unstable, which is of course not what we want. The "easy" > way out would be buying faster hardware, but that is usually not an > option ;-) Some upstreams, mostly media related but also unsuspectable like MySQL, use and test their apps with high optimizations. As an effect some of these apps tend to be _more_ stable with those high optimizations. If I recall correctly Ned Ludd (solar) did some work on having per package defined CFLAGS, don't know what was the intent of that work but integrate in portage a /etc/portage/package.env support, and let the packages mantainer _suggest_ optimal C*FLAGS may increase both stability and performance. However this require _a lot_ of manpower, add maybe unmanageable complexity, in every stage of a package life, from writing the ebuild to the final stabilization. > > So ... what can be done to get the stable maximum out of your hardware? > > In my experience (x86 centric - do other arches have different > "problems"?) the following is stable, but not necessarily the optimum: > - don't overtweak CFLAGS. "-O2 -march=$your_cpu_family" seems to be on > average the best, -O3 is often slower and can cause bugs see ^^^ > - don't do anything with ASFLAGS, LDFLAGS. This causes weird random > breakage (e.g. LDFLAGS="-O1" causes prelink to fail with "absurd" > errors) and doesn't give a noticeable performance boost see ^^^ > - check that all IDE disks use DMA mode, otherwise they are limited to > ~16M/s with a huge CPU usage penalty. Sometimes (application-specific) > increasing the readahead with hdparm gives a huge throughput boost. having more than one disk or a lot of memory add very interesting addition, read raid 0 (stripe) or tmpfs for working data that does'nt need a backup fex: $PORTIR, /var/tmp ... > - kernel tweaks like preempt may increase the responsiveness of the > system, but often reduce throughput and may have unexpected sideeffects > like random audio stutter as well as random kernel crashes ;-) I've found that preemption with the new standard 250Hz of the kernel is suitable for mostly needs, however no server here has preemption enabled ;-) > - kernel tweaks like setting swappiness or using a different I/O > scheduler (CFQ, deadline) should help, but I'm not aware of any "real" > benchmarks except microbenchmarks (can create 1M files 10% faster! - > yes, but how does it behave with a normal workload?) what is a normal workload ? Define it and creating tests should not be so difficult, then there are apps that can help to profiling, thinking at bootchart, sysproof, memproof, valgrind ... strace > - using a "smarter" filesystem can dramatically improve performance at > the potential cost of reliability. As data on FS reliability is hard to > find from unbiased sources this becomes a religious issue ... migrating > from ext3 to reiserfs makes "emerge sync" extremely much faster, but is > reiserfs sustainable? reiserfs is sustainable, at least for 99.999% of uses, last reiserfs bug on very high load (and with degraded raid5) is dated 4 years ago here. However upstream is going to the route of reiser4, much more complex, and much more unstable, latest problems where in 2.6.14, additionally no devs in gentoo are (will?) support it the patch for grub it's still not in place I think. > > Are there any application-specific tweaks (e.g. "use the prefork MPM > with apache2")? What is known to break things, what has usually > beneficial behaviour? Are there any useful benchmarks that show the > performance difference between different settings? is'n there "ab" [1] for apache testing ? Cheers, Francesco Riosa [1] http://httpd.apache.org/docs/2.0/programs/ab.html -- gentoo-dev@gentoo.org mailing list
[gentoo-dev] Optimizing performance
Hi all, I was wondering if there are any sane ways to optimize the performance of a Gentoo system. Overoptimization (the well known "-O9 -fomgomg" CFLAGS etc.) tends to make things unstable, which is of course not what we want. The "easy" way out would be buying faster hardware, but that is usually not an option ;-) So ... what can be done to get the stable maximum out of your hardware? In my experience (x86 centric - do other arches have different "problems"?) the following is stable, but not necessarily the optimum: - don't overtweak CFLAGS. "-O2 -march=$your_cpu_family" seems to be on average the best, -O3 is often slower and can cause bugs - don't do anything with ASFLAGS, LDFLAGS. This causes weird random breakage (e.g. LDFLAGS="-O1" causes prelink to fail with "absurd" errors) and doesn't give a noticeable performance boost - check that all IDE disks use DMA mode, otherwise they are limited to ~16M/s with a huge CPU usage penalty. Sometimes (application-specific) increasing the readahead with hdparm gives a huge throughput boost. - kernel tweaks like preempt may increase the responsiveness of the system, but often reduce throughput and may have unexpected sideeffects like random audio stutter as well as random kernel crashes ;-) - kernel tweaks like setting swappiness or using a different I/O scheduler (CFQ, deadline) should help, but I'm not aware of any "real" benchmarks except microbenchmarks (can create 1M files 10% faster! - yes, but how does it behave with a normal workload?) - using a "smarter" filesystem can dramatically improve performance at the potential cost of reliability. As data on FS reliability is hard to find from unbiased sources this becomes a religious issue ... migrating from ext3 to reiserfs makes "emerge sync" extremely much faster, but is reiserfs sustainable? Are there any application-specific tweaks (e.g. "use the prefork MPM with apache2")? What is known to break things, what has usually beneficial behaviour? Are there any useful benchmarks that show the performance difference between different settings? Thanks for your input, Patrick -- Stand still, and let the rest of the universe move signature.asc Description: This is a digitally signed message part
