Re: [gentoo-dev] The infinite git migration
11.06.2014 04:48, Duy Nguyen пишет: On Wed, Jun 11, 2014 at 5:59 AM, Patrick Lauer patr...@gentoo.org wrote: Another part: Git wasn't ready. The first migration attempt failed after consuming nearly 100GB of RAM! When it did work it took obscene amounts of time, and the result was unusably large (e.g. initial checkout would take 16GB RAM on the server, thus not allowing a few hundred devs to do checkouts the same day). The current state is almost usable, but it is still obscenely slow (e.g. initial clone taking ~10 CPU-minutes just to figure out what to do), but we can just throw more hardware at it. (10 minutes @ 3.6Ghz, so on my notebook it'll take about 4h to just clone the friggin repository. Too awesome!) Since v1.9.0 we can clone from a shallow repository. We can host two repos on the server: a full repo and a shallow one, containing history of only last year. Most of the time spent in initial clone is to verify the history. Shorter history would shorten that time. But you need to try out to see how long it actually is. I'm not sure if that 16GB includes cloning, or just plain checkout. If the latter, Git has a problem. Not sure if you can commit into that shallow repo(IIRC, you can not). I thought shallow clones is more suitable for users that want some state but do not want the whole history. -- Best regards, Sergey Popov Gentoo developer Gentoo Desktop Effects project lead Gentoo Qt project lead Gentoo Proxy maintainers project lead signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Creating a USE_EXPAND for ssl providers
Dear all, I'm a bit late to the party, but here is my $0.02: REQUIRED_USE= curl_ssl_winssl? ( elibc_Winnt ) ssl? ( ^^ ( [...] ) ) I don't like this. If the user specifies several SSL providers in make.conf, it should mean that any of these is fine and the ebuild can choose an arbitrary one. The exactly-one-of operator would cause emerge to complain in this case and possibly force the user to have complex package.use setups. With the number of ssl providers growing, like libressl, and with issues like bug #510974, I think its time we consider making this a uniform way of dealing with ssl providers in gentoo. We would proceed something like this: 1. Introduce a new USE_EXPAND called SSL which mirrors CURL_SSL --- becuase CURL_SSL is too provincial a name. 2. migrate curl and all its dependencies to the SSL use expand. 3. Migrate over all consumers of ssl to the new SSL use expand system. What do people think? I think a better name for the USE_EXPAND would be CRYPTO_PROVIDER (or similar) instead of just SSL, as the libraries are not strictly used for SSL but also for other forms of crypto (e.g. [1]). Best regards, Chí-Thanh Christopher Nguyễn [1] https://bugs.gentoo.org/show_bug.cgi?id=512664
Re: [gentoo-dev] Creating a USE_EXPAND for ssl providers
On 06/11/14 07:12, Chí-Thanh Christopher Nguyễn wrote: Dear all, I'm a bit late to the party, but here is my $0.02: REQUIRED_USE= curl_ssl_winssl? ( elibc_Winnt ) ssl? ( ^^ ( [...] ) ) I don't like this. If the user specifies several SSL providers in make.conf, it should mean that any of these is fine and the ebuild can choose an arbitrary one. The exactly-one-of operator would cause emerge to complain in this case and possibly force the user to have complex package.use setups. That's a good point and not one that I wasn't aware of. But how would we better design this? The only thing I can thing of (suggested earlier) is an eclass with some intelligence. I'm not sure of the most userfriendly way of doing this. With the number of ssl providers growing, like libressl, and with issues like bug #510974, I think its time we consider making this a uniform way of dealing with ssl providers in gentoo. We would proceed something like this: 1. Introduce a new USE_EXPAND called SSL which mirrors CURL_SSL --- becuase CURL_SSL is too provincial a name. 2. migrate curl and all its dependencies to the SSL use expand. 3. Migrate over all consumers of ssl to the new SSL use expand system. What do people think? I think a better name for the USE_EXPAND would be CRYPTO_PROVIDER (or similar) instead of just SSL, as the libraries are not strictly used for SSL but also for other forms of crypto (e.g. [1]). Agreed. Best regards, Chí-Thanh Christopher Nguyễn [1] https://bugs.gentoo.org/show_bug.cgi?id=512664 -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] Re: The state and future of the OpenRC project
On Wed, Jun 11, 2014 at 12:17 AM, Patrick Lauer patr...@gentoo.org wrote: Now git has some/all of the needed features, and people wait on a future potential git migration instead of figuring out the important bits now (a good part of that is defined in GLEP 63, but there's no action apart from work on gentoo-keys.I don't have motivation to try pusing this again this year ...) Well, the folks doing the git migration are basically waiting for the infra pieces to be done. A challenge here is that nobody has time, and sometimes the argument gets made that there is no sense doing A because you can't use it without B. The reality is that we need A, B, C, D, E to cut over, and none of them are useful on their own, so if everybody makes this argument we take no action. The actual migration and git tools should be fine now. Rich
Re: [gentoo-dev] The infinite git migration
On 10/06/14 06:59 PM, Patrick Lauer wrote: [snip] https://bugs.gentoo.org/show_bug.cgi?id=333531 The current state is almost usable, but it is still obscenely slow (e.g. initial clone taking ~10 CPU-minutes just to figure out what to do), but we can just throw more hardware at it. https://bugs.gentoo.org/show_bug.cgi?id=333685: git 2.0 has pack-bitmap apparently :) so once infra lands that that's basically the end of the major blockers afaik On 10/06/14 06:59 PM, Patrick Lauer wrote: Another part: Few people thought about the difficult problems in the migration. For example the rsync mirrors, which will still be used - the scripts that generate those will need to be changed, tested, and then replaced if a migration ever happens. https://bugs.gentoo.org/show_bug.cgi?id=333701: What comment #2 should have said: This bug is so low priority to the overall initiative that there shouldn't be anyone considering it a blocker, show me the git repo then we can talk :) signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] The state and future of the OpenRC project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 10/06/14 18:45, Andrew Savchenko wrote: Why are you saying that git is inefficient with large projects? Because it is. It was developed with efficiency in mind in the first place. Not for big projects. And kernel guys will likely disagree with git is not great with crazy big projects statement Not the last time I chatted to any of them, nor the last time I saw Linus say anything about this. And with git kernel bisection and other complicated (in terms or data manipulation) operations are quite fast even on old hardware. The Linux repository is nowhere near Portage-big. Rich mentions one of the reasons. Another is that for Linux, they very carefully decided to throw away history to make migration easier. I don't think we're about to throw away history for Portage. See also the Facebook link Julian posted. As Sergei linked, Facebook solved this by not using git. Personally I tend to believe that if git can't handle your project, you are doing something wrong in terms of modularity. But in the real world, that's not always amendable. - -- Alexander berna...@gentoo.org https://secure.plaimi.net/~alexander -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlOYUrAACgkQRtClrXBQc7XyoQD6AhPbhXqL4D7chnYnNF28+wGK oy+7MuNwiSgeQRCZk2EBAImLz2m2lEeMLrnGqUR+EKiROo+yWQhWPZ8tqCmB0kPO =PYpt -END PGP SIGNATURE-
Re: [gentoo-dev] Creating a USE_EXPAND for ssl providers
Dnia 2014-06-11, o godz. 13:12:38 Chí-Thanh Christopher Nguyễn chith...@gentoo.org napisał(a): REQUIRED_USE= curl_ssl_winssl? ( elibc_Winnt ) ssl? ( ^^ ( [...] ) ) I don't like this. If the user specifies several SSL providers in make.conf, it should mean that any of these is fine and the ebuild can choose an arbitrary one. The exactly-one-of operator would cause emerge to complain in this case and possibly force the user to have complex package.use setups. Your idea comes with three significant drawbacks: 1. USE flag setups become unclear -- user sees five different SSL providers turned on and has no clue which one is actually used. 2. You create 2^n-1 valid USE flag combinations which map to n different file sets. This means that there are 2^n-n-1 useless USE flag combination which make matching binary packages a PITA. 3. There is no clean way of enforcing SSL provider match between packages. Wasn't this thread initially about curl and rtmpdump requiring matching flags? -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [RFC] News item: GCC 4.8.3 defaults to -fstack-protector
On Tue, 10 Jun 2014 21:47:50 -0600 Ryan Hill rh...@gentoo.org wrote: v2: Restrict by arch -- Title: GCC 4.8.3 defaults to -fstack-protector Author: Ryan Hill rh...@gentoo.org Content-Type: text/plain Posted: 2014-06-10 Revision: 1 News-Item-Format: 1.0 Display-If-Installed: =sys-devel/gcc-4.8.3 Display-If-Keyword: amd64 Display-If-Keyword: arm Display-If-Keyword: mips Display-If-Keyword: ppc Display-If-Keyword: ppc64 Display-If-Keyword: x86 Beginning with GCC 4.8.3, Stack Smashing Protection (SSP) will be enabled by default. The 4.8 series will enable -fstack-protector while 4.9 and later enable -fstack-protector-strong. Will bug #332823 and its ilk somehow be mitigated? Emerging glibc with -fstack-protector still leads to similar problems. There doesn't currently seem to be a bug report about this that isn't marked INVALID. jer
Re: [gentoo-dev] Creating a USE_EXPAND for ssl providers
Michał Górny schrieb: I don't like this. If the user specifies several SSL providers in make.conf, it should mean that any of these is fine and the ebuild can choose an arbitrary one. The exactly-one-of operator would cause emerge to complain in this case and possibly force the user to have complex package.use setups. Your idea comes with three significant drawbacks: 1. USE flag setups become unclear -- user sees five different SSL providers turned on and has no clue which one is actually used. Does that really matter? What I got from bug 512664 comment 0 is the desire to ensure that a certain crypto provider (in this case, openssl) is *not* used. 2. You create 2^n-1 valid USE flag combinations which map to n different file sets. This means that there are 2^n-n-1 useless USE flag combination which make matching binary packages a PITA. This is indeed a problem, if producer and consumer of the binpkg are different entities. It could be mitigated by pre-populating the default list of crypto providers so that most users will not need to change it. 3. There is no clean way of enforcing SSL provider match between packages. Wasn't this thread initially about curl and rtmpdump requiring matching flags? It could be enforced if an eclass does the actual choice of crypto provider in a reproducible way. That would be not trivial though. Best regards, Chí-Thanh Christopher Nguyễn
Re: [gentoo-dev] The infinite git migration
On Wed, Jun 11, 2014 at 4:38 PM, Sergey Popov pinkb...@gentoo.org wrote: 11.06.2014 04:48, Duy Nguyen пишет: On Wed, Jun 11, 2014 at 5:59 AM, Patrick Lauer patr...@gentoo.org wrote: Another part: Git wasn't ready. The first migration attempt failed after consuming nearly 100GB of RAM! When it did work it took obscene amounts of time, and the result was unusably large (e.g. initial checkout would take 16GB RAM on the server, thus not allowing a few hundred devs to do checkouts the same day). The current state is almost usable, but it is still obscenely slow (e.g. initial clone taking ~10 CPU-minutes just to figure out what to do), but we can just throw more hardware at it. (10 minutes @ 3.6Ghz, so on my notebook it'll take about 4h to just clone the friggin repository. Too awesome!) Since v1.9.0 we can clone from a shallow repository. We can host two repos on the server: a full repo and a shallow one, containing history of only last year. Most of the time spent in initial clone is to verify the history. Shorter history would shorten that time. But you need to try out to see how long it actually is. I'm not sure if that 16GB includes cloning, or just plain checkout. If the latter, Git has a problem. Not sure if you can commit into that shallow repo(IIRC, you can not). Not before v1.9.0. Since 1.9, shallow repos are no different than full ones. You can fetch from a shallow repo, to a shallow repo, as well as push from/to a shallow repo. -- Duy
Re: [gentoo-dev] Creating a USE_EXPAND for ssl providers
Dnia 2014-06-11, o godz. 15:30:26 Chí-Thanh Christopher Nguyễn chith...@gentoo.org napisał(a): 3. There is no clean way of enforcing SSL provider match between packages. Wasn't this thread initially about curl and rtmpdump requiring matching flags? It could be enforced if an eclass does the actual choice of crypto provider in a reproducible way. That would be not trivial though. No, it can't. Let's say package A depends on package B and requires the same SSL provider. A has 'openssl gnutls' B has 'openssl gnutls polarssl' Now let's say the eclass prefers polarssl over the other two. How are you going to make A dep on B? -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] Creating a USE_EXPAND for ssl providers
Michał Górny schrieb: Dnia 2014-06-11, o godz. 15:30:26 Chí-Thanh Christopher Nguyễn chith...@gentoo.org napisał(a): 3. There is no clean way of enforcing SSL provider match between packages. Wasn't this thread initially about curl and rtmpdump requiring matching flags? It could be enforced if an eclass does the actual choice of crypto provider in a reproducible way. That would be not trivial though. No, it can't. Let's say package A depends on package B and requires the same SSL provider. A has 'openssl gnutls' B has 'openssl gnutls polarssl' Now let's say the eclass prefers polarssl over the other two. How are you going to make A dep on B? It is not trivial, but I don't think it is impossible. I had thought of the following, but have not carefully checked that it covers all cases. crypto-providers.eclass would have a list CRYPTO_PROVIDERS_SUPPORTED sorted descending by priority, and A and B would pass in a variable CRYPTO_PROVIDERS the acceptable providers. The eclass would provide functions which expand into USE dependencies to ensure that no higher-prioritized crypto provider is selected in B. Example: crypto-providers.eclass: CRYPTO_PROVIDERS_SUPPORTED=polarssl openssl gnutls libgcrypt libnettle ... crypto-providers_only() returns USE dependency on its arguments, and negative USE dependencies for all providers with higher priority, e.g. crypto-providers_only(gnutls) returns -crypto_providers_polarssl(-) -crypto_providers_openssl(-) crypto_providers_gnutls(-) crypto-providers_match(packagename) returns priority nested USE conditionals for all CRYPTO_PROVIDERS that can be fed into DEPEND, e.g. crypto_providers_match(B) would return crypto-providers_openssl? ( B[$(crypto_providers-only(openssl)] ) !crypto-providers_openssl? ( crypto-providers_gnutls? ( B[$(crypto_providers-only(gnutls)] ) ) A.ebuild CRYPTO_PROVIDERS=openssl gnutls DEPEND=$(crypto-providers_match(B)) B.ebuild CRYPTO_PROVIDERS=openssl gnutls polarssl Best regards, Chí-Thanh Christopher Nguyễn
Re: [gentoo-dev] The infinite git migration
On Wed, Jun 11, 2014 at 8:54 AM, Alex Xu alex_y...@yahoo.ca wrote: https://bugs.gentoo.org/show_bug.cgi?id=333701: What comment #2 should have said: This bug is so low priority to the overall initiative that there shouldn't be anyone considering it a blocker, show me the git repo then we can talk :) We have a git repo now. We can generate one at any time. We still don't have infra tools. I don't know if the repo is published anywhere, but there are plenty of bundles on dev.gentoo.org:/space/git-work/ If the infra backend scripts are actually published somewhere where an outside contributor can work on them and somebody is willing to do so, I'd be more than happy to push a bundle to github, or post it somewhere to download. If people don't want to work on them, fine. This is a volunteer effort. However, there really isn't any aspect of the git migration that isn't worth working on. If everybody is just going to sit around until everybody else does their part of the task, then we might as well give up now. :) So, I find stuff like comment #2 unhelpful, but at the same time nobody is really obligated to help out here. I just think that the infra components are certainly worth working on. Rich
Re: [gentoo-dev] The infinite git migration
On Wed, Jun 11, 2014 at 10:34 AM, Rich Freeman ri...@gentoo.org wrote: We have a git repo now. We can generate one at any time. We still don't have infra tools. I don't know if the repo is published anywhere, but there are plenty of bundles on dev.gentoo.org:/space/git-work/ So, if anybody does want to play around with Gentoo in git: https://github.com/rich0/gentoo-gitmig-2014-02-21.git This is just a snapshot from Feb. As a side note, it took about 10min to push to github, then git sat around waiting to terminate for 7min more before the remote side dropped the connection. At that point it appeared to have failed, but a few hours later everything was there. I'm sure the admins over at github are cursing my name. We need to get all the migration utility code published as well. My validation scripts are on github, with instructions on the wiki. Rich
[gentoo-portage-dev] Next team meeting
The next team meeting scheduler: http://whenisgood.net/xnq98br the results: http://whenisgood.net/xnq98br/results/9zqwgaj On the agenda: Team lead election, also co-lead, release co-ordinator next release plugin-sync finalization report and a couple decisions to make. repoman rewrite: round one breakup has started subslots, backtracking, resolver == we really need to get a sub team together to to handle the bugs, learn the code, create a new code model to work towards implementing. Sebastian is all alone on this. He needs help. -- Brian Dolbec dolsen
