Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Monday 13 January 2014 09:53:45 Tom Wijsman wrote: On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström wrote: At the end of the day we have one codebase which is engineered and another which has evolved. Too broad generalization, too much assumption; both can be held as meaning nothing compared to what engineered and evolved could really be, but as with doing that, it gets a subjective nature. In other words, the lack of context makes this statement meaningless. anyone who has spent serious time in the portage code base knows it sucks. i'm not blaming anyone -- it's no one's fault. portage started as prototyped idea that has since had more and more stuff piled onto it over the years by each successive maintainer. devs i've talked to agree that it sucks to work with. it's why pkgcore was born in the first place. i'd like to see portage pkgcore merge, but it'd take quite a bit of work on the portage side to migrate step by step. we generally haven't had leads who have enough time sorting out the existing bugs/feature requests to try and also restructure/reshape things. maybe by trying to get new interest in the project means we can find some people willing to rip off some sizable chunks. the fact that the public API is pretty much non-existent is nice because it means we're free to change/break whatever we want. note though that the let's rewrite everything in a branch and then merge later approach doesn't work. it's been done a few times in portage land and aborted each time. it's rare for this to work for other projects either. small steps are much easier to review/merge. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On 01/13/14 03:43 PM, Alexander Berntsen wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 13/01/14 09:39, C. Bergström wrote:
Drive-by trolling comment but I wish the effort to keep porkage
alive would have instead been directed towards pkgcore.
Realistically, we have to keep updating them both in parallel. pkgcore
needs to be brought up to portage-level functionality, and we need to
keep fixing portage bugs for its many users. We could have a technical
discussion on the merits of pkgcore vs. portage, but in the end, it's
about the users.
For the record, I would be very happy if you hacked on pkgcore. Just
as happy as if you hacked on portage, even. So please do. :-)
Where I work uses pkgcore[1], but not the areas which are generally
beneficial to the whole community. (We use it as part of a web
application to handle testsuites which have build dependencies.) We can
blah blah about performance of resolving package dependencies all day
long, but it's clearly not a game changer feature for users. (or they
just don't know) Long term the API to pkgcore could be beneficial, but
again I'm not sure it's a game changer for users. Dare I turn this from
a trolling comment into a laundry list of pros/cons of pkgcore and why
now may be a good time to invest in the future. At the end of the day we
have one codebase which is engineered and another which has evolved.
If gentoo only ever wants to fetch tarballs and build source.. who
cares.. If you ever want to build something on top of that - you will
painfully realize that there's only 1 choice.
On 01/13/14 03:59 PM, Dirkjan Ochtman wrote:
If you know your email is going to be drive-by trolling, maybe just
hold it in next time?
Yeah I know better, but this time was like a fart - (half attempt joking)
[1] /* Side details */
In a nutshell we make it possible to track the results of make check
or equivalent test coverage for every source package. There's a little
work involved for setting up each package, but it gives the easy ability
to *know* and prove that between xyz ${FLAGS} there's no regressions.
For example: How do you *know* that fftw or ssl is regression free when
you enable avx, fma or some other higher level of optimization (-O3). By
knowing I don't mean just result correctness, but also potentially
runtime performance, code size and potentially compile times. (If those
are things you care about)
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, Jan 13, 2014 at 10:15 AM, C. Bergström cbergst...@pathscale.com wrote: On 01/13/14 03:43 PM, Alexander Berntsen wrote: Where I work uses pkgcore[1], but not the areas which are generally beneficial to the whole community. (We use it as part of a web application to handle testsuites which have build dependencies.) We can blah blah about performance of resolving package dependencies all day long, [...] Not sure about what you mean with blah blah. But given the amount of both disk caches (metadata, vdb cache) and memory caches (the in-memory aux_db cache that portage loads using pickle (it's a dict) takes like 70-100Mb of RAM on an average desktop system), Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Ironically, launching the same emerge command twice, will take more or less the same time. Yeah, this is probably bad design... -- Fabio Erculiani
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On 01/13/14 04:31 PM, Fabio Erculiani wrote: On Mon, Jan 13, 2014 at 10:15 AM, C. Bergström cbergst...@pathscale.com wrote: On 01/13/14 03:43 PM, Alexander Berntsen wrote: Where I work uses pkgcore[1], but not the areas which are generally beneficial to the whole community. (We use it as part of a web application to handle testsuites which have build dependencies.) We can blah blah about performance of resolving package dependencies all day long, [...] Not sure about what you mean with blah blah. But given the amount of both disk caches (metadata, vdb cache) and memory caches (the in-memory aux_db cache that portage loads using pickle (it's a dict) takes like 70-100Mb of RAM on an average desktop system), Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Ironically, launching the same emerge command twice, will take more or less the same time. Yeah, this is probably bad design... ack - I know the benefits (and downsides) of pkgcore compared to portage, but I leave that up to others who would like to voice their opinion. It would be great to get pkgcore up to feature parity with portage, but I don't have the resources to help with that. (In the future, possibly next month, I will try to put some bounties)
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. Yes, a side effect of not having that is that a lot of tools would need to be rewritten too if we would make a sudden move to pkgcore; or at least, it would mean that Portage would still need to be around to serve as an API, but that would also mean it still needs to be maintained. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: At the end of the day we have one codebase which is engineered and another which has evolved. Too broad generalization, too much assumption; both can be held as meaning nothing compared to what engineered and evolved could really be, but as with doing that, it gets a subjective nature. In other words, the lack of context makes this statement meaningless. What we're looking for is a well-engineered _and_ well-evolving distribution, not just a well-engineered _or_ well-evolving package; and thus there's a lot more to it to really use this as an argument. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 13/01/14 09:46 AM, Tom Wijsman wrote: On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. ..you mean, define the API and include it in PMS? how much bureaucracy will that need to occur? -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlLT/o0ACgkQ2ugaI38ACPBIlwD/UhmM7jiu0c3kbczm/p0eA0nE QklKzRWkYxLpUpWEwegA/0GzTsFmUL/9C+xVvh1rpKjk23L6LAR/mWuiB34Vmnb0 =vpX0 -END PGP SIGNATURE-
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 10:31:56 +0100 Fabio Erculiani lx...@gentoo.org wrote: Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Half a minute if you disable backtracking which you don't need. :) Ironically, launching the same emerge command twice, will take more or less the same time. Determinism results in more or less the same time, that's correct; proper benchmarks would show you a similar result. Yeah, this is probably bad design... Too broad generalization with no fundamental argument about the design. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, 13 Jan 2014 09:56:13 -0500 Ian Stakenvicius a...@gentoo.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 13/01/14 09:46 AM, Tom Wijsman wrote: On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. ..you mean, define the API and include it in PMS? how much bureaucracy will that need to occur? The API can be separate of the PMS. The bureaucracy needed for an API to provide to tools, that doesn't need as much involvement as PMS. - -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJS1AbYAAoJEJWyH81tNOV90DEIAIk+zQe4vgJ2XWchQLET064Q mij/sQQRcI52PZwXFfRh23G+dKkywsPIg+US5KmbWJTtT4UbbLFnn4KCRB/QZE6E 43wHYsKGMgTovpkXQZEy8O4GbFUOb1k9DrsR05r91+QUUUmxxxi9ODq86kavtPVs UaPyz3me4j3olHLSxzUtxJd5/AgHGwkH+YHtcWny2oaXNUIFpusIPRURqyFyE4xI Leq/w8/nsmeCc25pPN4cbY6bBwOwCxsZZwik3gmz6FzrAVwCjuRvX8n7szl8viNr Kd56odFjX/n7pAL+Rj52spz6TNcSe1H8IMkwI9rMYc/LYQSw+/uCGnAXeIxrzWY= =F2d3 -END PGP SIGNATURE-
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 15:58:13 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 10:31:56 +0100 Fabio Erculiani lx...@gentoo.org wrote: Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Half a minute if you disable backtracking which you don't need. :) Which sadly also means that some updates get skipped silently. (Those which would trigger rebuilds of other packages because of sub-slot deps, had that case yesterday). Ironically, launching the same emerge command twice, will take more or less the same time. Determinism results in more or less the same time, that's correct; proper benchmarks would show you a similar result. I guess he means that the (according to the file sizes) extensive caching doesn't seem to be of much use. -- Luis Ressel ara...@aixah.de GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 16:38:59 +0100 Luis Ressel ara...@aixah.de wrote: On Mon, 13 Jan 2014 15:58:13 +0100 Tom Wijsman tom...@gentoo.org wrote: Half a minute if you disable backtracking which you don't need. :) Which sadly also means that some updates get skipped silently. (Those which would trigger rebuilds of other packages because of sub-slot deps, had that case yesterday). Can you give an example of that? Rebuilds don't cause a different solution in the graph afaik; so, I wouldn't see how that would form a big problem. I also think this would still be covered by preserved-rebuild and/or revdep-rebuild afterwards. In any case, I've not been experiencing problems with this; but am interesting to know how this could go wrong, as no backtracking works for me and I hope it will continue to do so. Btw, this also makes me question how proper pkgcore can do rebuilds. Ironically, launching the same emerge command twice, will take more or less the same time. Determinism results in more or less the same time, that's correct; proper benchmarks would show you a similar result. I guess he means that the (according to the file sizes) extensive caching doesn't seem to be of much use. For that, _all_ caching would need to be removed before the first run; when there is a mention of the same, I doubt this was done at all. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, Jan 13, 2014 at 7:38 AM, Luis Ressel ara...@aixah.de wrote: On Mon, 13 Jan 2014 15:58:13 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 10:31:56 +0100 Fabio Erculiani lx...@gentoo.org wrote: Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Half a minute if you disable backtracking which you don't need. :) Which sadly also means that some updates get skipped silently. (Those which would trigger rebuilds of other packages because of sub-slot deps, had that case yesterday). Ironically, launching the same emerge command twice, will take more or less the same time. Determinism results in more or less the same time, that's correct; proper benchmarks would show you a similar result. I guess he means that the (according to the file sizes) extensive caching doesn't seem to be of much use. The caching may not be of use, depending on your configuration. (For example, if you use a gentoo-x86 checkout as your main repo, you will probably want to run generate cache entries whenever you cvs up.) It is there to cache ebuild metadata, because if your depgraph has a few thousand nodes, having to spawn bash to generate the metadata for every node is very expensive. That being said, for most users that use rsync, the metadata is pre-generated. As long as they are not changing the cache indicators (not sure if this is mtime or md5sum these days) they should be seeing a benefit. This is not meant to imply that portage is always fast; there are plenty of other modules (such as the aforementioned backtracking) that can take a long time to find solutions. That is partially why you see Tomwij turning off that feature. It is helpful for users cause it can automatically find solutions for users that are otherwise unsolvable (and thus avoids the user having to find a solution to the depgraph manually.) -A -- Luis Ressel ara...@aixah.de GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 16:46:08 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 16:38:59 +0100 Luis Ressel ara...@aixah.de wrote: On Mon, 13 Jan 2014 15:58:13 +0100 Tom Wijsman tom...@gentoo.org wrote: Half a minute if you disable backtracking which you don't need. :) Which sadly also means that some updates get skipped silently. (Those which would trigger rebuilds of other packages because of sub-slot deps, had that case yesterday). Can you give an example of that? Rebuilds don't cause a different solution in the graph afaik; so, I wouldn't see how that would form a big problem. I also think this would still be covered by preserved-rebuild and/or revdep-rebuild afterwards. No, the problem wasn't that rebuilds weren't done (btw: this is not about @preserved-rebuilds, but about subslot dependencies), but that updates which would trigger such rebuilds are silently ignored. This happened to me yesterday while trying --backtrack=0. The available update to dev-haskell/parsec simply didn't show up (haskell ebuilds make heavy use of subslot deps), I only noticed this because I knew there was in fact an update available (thanks to eix-diff). Only after enabling backtracking Portage found the update. This might well be a bug, perhaps I'll examine the situation when I've got more time. -- Luis Ressel ara...@aixah.de GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, Jan 13, 2014 at 5:49 PM, Alec Warner anta...@gentoo.org wrote: [...] This is not meant to imply that portage is always fast; there are plenty of other modules (such as the aforementioned backtracking) that can take a long time to find solutions. That is partially why you see Tomwij turning off that feature. It is helpful for users cause it can automatically find solutions for users that are otherwise unsolvable (and thus avoids the user having to find a solution to the depgraph manually.) Yeah, correct. But it would be nice if Portage backtrack_depgraph() would memoize (asynchronously serializing to disk, for instance) partial results for later use. AFAIR, last time I checked, it wasn't doing that. Also, it would be nice if the aux_db cache of the vdb could be stored in a sqlite3 db rather than in RAM. This dict is consuming a huge cut of memory for little reason (a single vdb.dbapi.match() can eat 100MB of RAM). We know how Python deals with freed memory... Sigh... -A -- Luis Ressel ara...@aixah.de GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD -- Fabio Erculiani
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, Jan 13, 2014 at 04:15:37PM +0700, C. Bergström wrote: At the end of the day we have one codebase which is engineered and another which has evolved. I'll take an evolved codebase over engineered anyday. You do realize that is exactly why Linux has succeeded, right? The kernel has evolved, and was never engineered. There's lots people should be learning from biology... So you are using the benifits of evolution right now on your system, don't knock it, it's proven to work. greg k-h
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Tue, Jan 14, 2014 at 12:42:00AM +0700, C. Bergström wrote: On 01/14/14 12:37 AM, Greg KH wrote: On Mon, Jan 13, 2014 at 04:15:37PM +0700, C. Bergström wrote: At the end of the day we have one codebase which is engineered and another which has evolved. I'll take an evolved codebase over engineered anyday. You do realize that is exactly why Linux has succeeded, right? The kernel has evolved, and was never engineered. There's lots people should be learning from biology... So you are using the benifits of evolution right now on your system, don't knock it, it's proven to work. I'll bite - While I don't think nature stopped to properly design interfaces along the way. I bet you Linus wouldn't agree with your comment very much. I don't think you have been paying attention much, I'm directly quoting Linus: Linux is evolution, not intelligent design - Linus There are many more statements exactly like this from him over the years, do a bit of research to dig them up. 1) I expect quite a bit of time has gone into (Solaris and Linux) kernel interfaces Time doesn't mean they haven't evolved. 2) Any larger or invasive changes require quite a bit of planning, review and testing. (Possibly with tests (public/private) to cover a large amount of the new/existing feature And we always get it wrong, so they evolve into something that later on works. Seriously, this is how the very system you are using has been created, it's a well-documented fact (look at our changelogs for details.) sorry, greg k-h
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On 01/14/14 12:37 AM, Greg KH wrote: On Mon, Jan 13, 2014 at 04:15:37PM +0700, C. Bergström wrote: At the end of the day we have one codebase which is engineered and another which has evolved. I'll take an evolved codebase over engineered anyday. You do realize that is exactly why Linux has succeeded, right? The kernel has evolved, and was never engineered. There's lots people should be learning from biology... So you are using the benifits of evolution right now on your system, don't knock it, it's proven to work. I'll bite - While I don't think nature stopped to properly design interfaces along the way. I bet you Linus wouldn't agree with your comment very much. 1) I expect quite a bit of time has gone into (Solaris and Linux) kernel interfaces 2) Any larger or invasive changes require quite a bit of planning, review and testing. (Possibly with tests (public/private) to cover a large amount of the new/existing feature To clarify - by engineering I mean significant forethought into the design and implementation. I'm not trying to take away the achievements of any particular piece of software. In this case it just one of those you know it when you see it. If I wasn't clear or you don't understand what I meant - that's ok because I meant no offense to anyone.
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 16:46:08 +0100 Tom Wijsman tom...@gentoo.org wrote: Rebuilds don't cause a different solution in the graph afaik; so, I wouldn't see how that would form a big problem. I also think this would still be covered by preserved-rebuild and/or revdep-rebuild afterwards. There used to be a feature whereby if resolution took too long, you'd get an incomplete answer. The second time you ran the resolution, the time could differ, so you'd get a different incomplete answer. Dunno if this has been fixed yet. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 2014-01-13 at 15:46 +0100, Tom Wijsman wrote: On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. I have been working towards that for years, but, things keep getting in the way. By things, I mean other project work that needs to be done as well. I got started in all this working on porthole, which uses portage's api for much of it's information. To get more features, information, I wanted to be able to incorporate some of gentoolkit's info. Namely equery, but most of it's working code was embedded with it's output. I worked hard at separating out it's working code from it's output so it would be usable by other tools. I have also been working on making a pkgcore backend for porthole. In doing that it required making a different backend for portage to get some uniformity for the frontend. I have had to put development of those on hold, due to taking over layman's development. I gave it a new high level api, modified it's mid level code and gave it a nice api that can be used by other apps, guis, etc.. One of the things that came up with layman was to be able to gpg enable it to verify it's repositories.xml list it downloads. I did that. In so doing created dev-python/pyGPG a python lib, which has now brought me in to developing Gentoo-keys (another project that could use help) to manage gentoo's gpg release, developer keys, and verification. Also at the same time a year ago, there was a lot of talk about moving the default location of the gentoo tree, but it could not be done with current catalyst code. So offered to help out as that project was severely lacking people with decent python skills. That code base is like what portage code was 8 years ago. And if you thought todays portage code is bad... you would cringe to see it's code from 8 years ago. Now portage was in trouble, while there were some people stepping in to fix things, I stepped up to help drive out a new release. I am now interim lead till we hold an election. So most of my time now is spent steering projects, more than coding. Hey, it's all work that has to be done. So I'm putting out fires where it needs it most. With that aside. One of the biggest hurdles new developers face with pkgcore is the lack of decent api docs and flow charts. Brian was brilliant and OCD about attaining speed, but at the cost of difficulty in following the code and creating the steep learning curve. I have been trying to get these other projects to a point I could create the docs and charts needed so that it would be easier for new developers to find their way in pkgcores code. That is when pkgcore will make more headway at becoming portage's replacement. But some new fires keep popping up. Long story in a nutshell, gentoo could use more GOOD firefighters. Sorry for the long speech ;) your friendly gentoo python firefighter -- Brian Dolbec dol...@gentoo.org signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 15:46:59 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. Not an API. APIs are bad. What we should have is a good set of lightweight Unix-friendly command line tools. See, for example, the Scripting Commands section of man cave. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 18:03:31 +0100 Luis Ressel ara...@aixah.de wrote: No, the problem wasn't that rebuilds weren't done (btw: this is not about @preserved-rebuilds, but about subslot dependencies), but that updates which would trigger such rebuilds are silently ignored. This happened to me yesterday while trying --backtrack=0. The available update to dev-haskell/parsec simply didn't show up (haskell ebuilds make heavy use of subslot deps), I only noticed this because I knew there was in fact an update available (thanks to eix-diff). Only after enabling backtracking Portage found the update. This might well be a bug, perhaps I'll examine the situation when I've got more time. That description is broad, it sounds like you didn't pass -uD to emerge; if it is part of the dependency graph (it should, otherwise this discussion would be useless) and is an update (it is, due to eix-diff), then with -uD it would pull that in. This even doesn't have to do anything with sub-slot dependencies as far as I know; because why would it check the sub-slot dependencies if it were an upgrade, the upgrade takes precedence over the sub-slot dependencies. The other option is that I am confused about this; as you can see, given the possible scenarios that could happen I might have a different one on mind than you do. If backtracking doesn't get to it, it would have to list it as a conflict; if it doesn't do that at the moment, then we certainly have a bug somewhere I think. Thank you in advance for investigating this. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 08:49:17 -0800 Alec Warner anta...@gentoo.org wrote: The caching may not be of use, depending on your configuration. (For example, if you use a gentoo-x86 checkout as your main repo, you will probably want to run generate cache entries whenever you cvs up.) It is there to cache ebuild metadata, because if your depgraph has a few thousand nodes, having to spawn bash to generate the metadata for every node is very expensive. It sounds crazy to use gentoo-x86 without a cache; to some extent, the earlier mentions on this ML by Ciaran that we should improve input would make more sense if would want to run without a cache. But I feel like this shouldn't be a topic of discussion unless we intent 'works fast on a plain CV checkout' to be an actual feature. I haven't checked, but doesn't Portage in that case build its own cache? It is helpful for users cause it can automatically find solutions for users that are otherwise unsolvable (and thus avoids the user having to find a solution to the depgraph manually.) If we document how to process --depgraph=0 output, I believe it could just as well be used; but yes, it basically involves having to run a few manual upgrades (emerge -1 ...) and/or masks to get going. But you can do those things in seconds rather than to wait a few minutes... Some of these Portage could even do automatically regradless, like for instance the 'no parents' slot conflicts; in which case I think you can just always upgrade to the newer version, instead of outputting them it put them as upgrades at the end of the emerge list I think. But I haven't investigated that further, I'm just getting tired of seeing 'em. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 18:05:21 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Mon, 13 Jan 2014 16:46:08 +0100 Tom Wijsman tom...@gentoo.org wrote: Rebuilds don't cause a different solution in the graph afaik; so, I wouldn't see how that would form a big problem. I also think this would still be covered by preserved-rebuild and/or revdep-rebuild afterwards. There used to be a feature whereby if resolution took too long, you'd get an incomplete answer. The second time you ran the resolution, the time could differ, so you'd get a different incomplete answer. Dunno if this has been fixed yet. We should document what Portage does; but from the word 'backtrack' I think it tries with what it is given, and if it doesn't succeed it goes one step back and tries to do it with more. But how exactly that is implemented, I think only people like Zac and Arfrever can tell us; or we read through the relevant code ourselves. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 19:16:45 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 08:49:17 -0800 Alec Warner anta...@gentoo.org wrote: The caching may not be of use, depending on your configuration. (For example, if you use a gentoo-x86 checkout as your main repo, you will probably want to run generate cache entries whenever you cvs up.) It is there to cache ebuild metadata, because if your depgraph has a few thousand nodes, having to spawn bash to generate the metadata for every node is very expensive. It sounds crazy to use gentoo-x86 without a cache; to some extent, the earlier mentions on this ML by Ciaran that we should improve input would make more sense if would want to run without a cache. But I feel like this shouldn't be a topic of discussion unless we intent 'works fast on a plain CV checkout' to be an actual feature. I think you're misunderstanding that comment. It's not just the file formats that are bad. It's also what data is provided. Two examples are the massive lengths we have to go to to figure out what to do with || dependencies (thanks to the horrific abuses commonly used in the tree...), and the well-documented REQUIRED_USE screwup. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 18:07:39 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Mon, 13 Jan 2014 15:46:59 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 16:15:37 +0700 C. Bergström cbergst...@pathscale.com wrote: Long term the API to pkgcore could be beneficial, but again I'm not sure it's a game changer for users. Long term, we should have an independent API backend that tools can query; not rewrite our tools every time users want to use them with a different package manager. Not an API. APIs are bad. What we should have is a good set of lightweight Unix-friendly command line tools. See, for example, the Scripting Commands section of man cave. It still is an API that way, just expressed differently; if you would only do this you're introducing forks where you might not need them. Providing shell commands is one possible binding to the API... -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 18:21:58 + Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Mon, 13 Jan 2014 19:16:45 +0100 Tom Wijsman tom...@gentoo.org wrote: On Mon, 13 Jan 2014 08:49:17 -0800 Alec Warner anta...@gentoo.org wrote: The caching may not be of use, depending on your configuration. (For example, if you use a gentoo-x86 checkout as your main repo, you will probably want to run generate cache entries whenever you cvs up.) It is there to cache ebuild metadata, because if your depgraph has a few thousand nodes, having to spawn bash to generate the metadata for every node is very expensive. It sounds crazy to use gentoo-x86 without a cache; to some extent, the earlier mentions on this ML by Ciaran that we should improve input would make more sense if would want to run without a cache. But I feel like this shouldn't be a topic of discussion unless we intent 'works fast on a plain CV checkout' to be an actual feature. I think you're misunderstanding that comment. It's not just the file formats that are bad. It's also what data is provided. Two examples are the massive lengths we have to go to to figure out what to do with || dependencies (thanks to the horrific abuses commonly used in the tree...), and the well-documented REQUIRED_USE screwup. Iotw, 'input'; but thank you for clarifying. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Mon, 13 Jan 2014 19:27:36 +0100 Tom Wijsman tom...@gentoo.org wrote: Not an API. APIs are bad. What we should have is a good set of lightweight Unix-friendly command line tools. See, for example, the Scripting Commands section of man cave. It still is an API that way, just expressed differently The term API seems to mean library or communication interface these days (see Git API and the like). So for clarity, it's probably better to make the distinction. if you would only do this you're introducing forks where you might not need them. Providing shell commands is one possible binding to the API... Forks are far less of a big deal than trying to share a process between programming languages. This is a *huge* deal when trying to integrate modern C++ with languages with primitive thread-hostile runtimes like Python and Ruby. Unless you're doing a heavy GUI (in which case portability probably isn't going to cut it anyway, without a far larger API than it's worth providing), most of the cost for most of the use cases for this is in the PM reading in configs and the like. The way around that is to allow communication via pipes, still in a Unix-friendly manner. This can be implemented nearly transparently. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On 01/13/2014 10:58 PM, Tom Wijsman wrote: On Mon, 13 Jan 2014 10:31:56 +0100 Fabio Erculiani lx...@gentoo.org wrote: Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Half a minute if you disable backtracking which you don't need. :) Or if you don't have a supercharged turbo-CPU ... no joke ... Half an hour. Yeah, on an E-350 it's GREAT FUN to WAIT for portage ;) It took me three days of frustration just to figure out an upgrade path.
Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)
On Tue, 14 Jan 2014 07:22:25 +0800 Patrick Lauer patr...@gentoo.org wrote: On 01/13/2014 10:58 PM, Tom Wijsman wrote: On Mon, 13 Jan 2014 10:31:56 +0100 Fabio Erculiani lx...@gentoo.org wrote: Portage can still take *minutes* to calculate the merge queue of a pkg with all its deps satisfied. Half a minute if you disable backtracking which you don't need. :) Or if you don't have a supercharged turbo-CPU ... no joke ... Half an hour. Well, in your case disabling it thus leads to a quite huge improvement. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
