[gentoo-dev] Re: [OT] pkgcore bikeshed (was Portage team)

[Steven J. Long]

On Mon, Jan 13, 2014, Ciaran McCreesh wrote:
> On Mon, 13 Jan 2014 19:27:36 +0100
> Tom Wijsman  wrote:
> > > Not an API. APIs are bad. What we should have is a good set of
> > > lightweight Unix-friendly command line tools. See, for example, the
> > > "Scripting Commands" section of "man cave".
> > 
> > It still is an API that way, just expressed differently
> 
> The term "API" seems to mean "library or communication interface" these
> days (see "Git API" and the like). So for clarity, it's probably better
> to make the distinction.

I think Tom was aware of that: hence "expressed differently". You appear
to have missed his point, which was not about something so simple and
well-known, judging by your inability to perceive it, and the state of
"modern" software.

> > if you would only do this you're introducing forks where you might
> > not need them. Providing shell commands is one possible binding to
> > the API...
> 
> Forks are far less of a big deal than trying to share a process between
> programming languages. This is a *huge* deal when trying to integrate
> modern C++ with languages with primitive thread-hostile runtimes like
> Python and Ruby.
> 
> Unless you're doing a heavy GUI (in which case portability probably
> isn't going to cut it anyway, without a far larger API than it's worth
> providing), most of the cost for most of the use cases for this is in
> the PM reading in configs and the like. The way around that is to allow
> communication via pipes, still in a Unix-friendly manner. This can be
> implemented nearly transparently.

Yes and the traditional Unix way to implement those shell commands is
to write a lib and then some apps as thin wrappers on top of it, and
possibly more complex ones on top of that as time goes by, and users
require. These components can in turn be used by other apps, or their
crappy non-C runtimes, as a basis for future development.

So Tom's point stands.

As for modern C++ being the be-all and end-all that sets the exemplar
.. LMAO. Good one. 30 years to reimplement LISP badly. Greenspun^N

I won't bother with your assertions about GUIs etc. Just don't presume
silence for agreement, as you are so wont to do. The obvious truths
about pipes etc, while true have very little to do with the point being
made. Still, glad you finally got hold of APUE.

*wipes tears from eyes and moves on*
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

[gentoo-dev] pkgcore EAPI-6 (Was: OT: pkgcore bikeshed)

[Steven J. Long]

On Mon, Jan 13, 2014, Alexander Berntsen wrote:
> > Updating both in parallel isn't hard: once pkgcore is up to EAPI-5,
> >  EAPI-6 isn't that much work (mostly bash afair.)
> If it is trivial: show us the code.

Ah that old canard. Tell you what: I hereby undertake to deliver
everything currently listed at [1] for pkgcore, once radhermit has
declared he's done on his round of work with pkgcore internals and
it's up to EAPI-5, as tested against ebuilds, ferringb has approved it,
and its portage formatter is good enough for update[2] to use it again,
on EAPI-5 ebuilds.

Sorry the latter is why I care, and I've always pushed for it, so
whether that annoys anyone else is not my problem. I will be involved
in testing that aspect, obviously, and may need to patch update for
other interactions with pmerge, so that will come first.

I will deliver that within a calendar month, or 30 days, of the above
happening, for QA and acceptance-testing, unless ferringb, radhermit
or dol-sen veto it, preferably in #pkgcore but see disclaimer at end.

To avoid confusion, in case more is added, I'll list them, so you can
hold me to my word later:

bash 4.2   #431340 
nonfatal die   #451938 -- truly lame to do this via a new func, imo
reads like working round the PMS, which /is/ insane. static libs in
/usr is nothing on that, when you think about it.
get_libdir #463586 Used in econf, but so far not available as
 separate PM function
Doc install function (edocs)  #459692 
 Allow DOCS=""  #463736 
 Directory support for DOCS  #481980 
Query function for IUSE  #449862
in addition, query IUSE_EFFECTIVE -- from [3] this is what should
be queried?
PATCHES support in default src_prepare  #463692 
Unpack .txz   #458102 
Case-fold extensions in unpack  #476730 
unpack() accept absolute paths  #483244 
epatch_user  - a sane implementation as we discussed 2 years ago or so
Source eclasses only once #422533 -- amended at [4]
EJOBS   #273101 - emake integration -- for other languages or build-
systems, the eclasses would have to handle it. They are not part
of the PM/ebuild.sh, so out of scope. (There is a qm as to whether
this is still desired 4 years later.)

non-bash, but i'll provide support script as and if req'd:
HDEPEND   #317337 -- CHOST dependencies req at build-time (yaf way
of adding a borked LDEPEND if you ask me, but w/e.)

I won't do failglob (#463822) as it will make everyone's life a lot
harder in the long run, and turn off anyone who actually has real BASH
experience. AFAIC we want them in the same way as we want expert
pythonistas. I'll happily add Kent's idea though, at least for this
setting; he's a guy you should all listen to a lot more, imnsho.

I'd do it like this:
failglob will apply to the function it appears in, and any callee
functions, and must be paired with reglob (or w/e) to restore, which
will also restore a nullglob invocation. (reglob must be called
on all return paths before function return, if you're doing something
complex. Most functions won't be though, ime.)

If ferringb and radhermit are ok with it, I'll implement it. If not,
it'll be trivial to import the crappy version from portage.

> > At that point, put portage into feature-freeze, and only bugfix it.
> >  Call a hiatus on new EAPIs for 6 months and put all effort into 
> > making damn sure pkgcore is a drop-in replacement.
> I don't see this happening as it stands right now. Let's revisit this
> when pkgcore is more up to date.

Yes, "at that point".

Now you may think I'm talking before I can implement, what I accused
someone else of; feel free to. Those are mostly trivial to a BASH
scripter, and I write (and throw away) more complex things practically
every day.

See [2] if you need verification, though it's not a work-project any
more, and was never meant to be when I started it as a training
assignment. We do use it at work though, so I get to maintain it.

And there is no way we could have done the ABI /etc/warning stuff, if
we hadn't had pkgcore to test resolves with. portage was just too slow,
when we were paired and needed to flow, to implement quite major
stuff in hindsight; ferringb even mentioned a need for exactly that
sort of mechanism in Gentoo, on the list last feb/mar i think it was,
when I was on hiatus. I remember being annoyed I'd missed the
discussion when I got back to FLOSS stuff.

Porcelain and plumbing.

Feel free to step up and pitch in too. The more the merrier, and the
lighter the load. Note there are queries on some of the above, and
no clear definition of what EAPI 6 is yet. So once you've made your
minds up about that, then you can start asking for your users to
implement it, I'd think. After all, devs are users too, and we've
all got other things to do.

TTFN,
steveL.

Disclaimer: I don't speak for pkgcore, or any of the people mentioned,
nor have I discussed any of this with them, so it's their call, now or
when they get to that stage. I

Re: [gentoo-dev] Default USE changes for fortran and mudflap?

[Mike Frysinger]

On Sunday 12 January 2014 02:53:47 Ryan Hill wrote:
> While I'm adding USE defaults to toolchain.eclass and moving them out of
> the profiles, I thought now would be a good time to review a couple
> default flag settings.
> 
> mudflap:
> This is currently enabled by default but I'd like to disable it.  It
> controls libmudflap and the -fmudflap flag.  I think the only reason this
> flag exists is so we can disable it in crossdev.  It's not required by
> anything in the tree, the code is bitrotten and has been removed for GCC
> 4.9. If you know how to use -fmudflap, you know how to set a USE flag.

it was the hot new checker when it was first added, but it never really caught 
on and asan is the way to go now.  i occurred to me a few weeks ago to turn it 
off by default after watching another cross-compile go by with it on, but looks 
like you got it covered.
-mike


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Tom Wijsman]

On Tue, 14 Jan 2014 07:22:25 +0800
Patrick Lauer  wrote:

> On 01/13/2014 10:58 PM, Tom Wijsman wrote:
> > On Mon, 13 Jan 2014 10:31:56 +0100
> > Fabio Erculiani  wrote:
> > 
> >> Portage can still take *minutes* to calculate the merge queue of a
> >> pkg with all its deps satisfied.
> > 
> > Half a minute if you disable backtracking which you don't need. :)
> 
> Or if you don't have a supercharged turbo-CPU ... no joke ...
> 
> Half an hour.

Well, in your case disabling it thus leads to a quite huge improvement.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Patrick Lauer]

On 01/13/2014 10:58 PM, Tom Wijsman wrote:
> On Mon, 13 Jan 2014 10:31:56 +0100
> Fabio Erculiani  wrote:
> 
>> Portage can still take *minutes* to calculate the merge queue of a
>> pkg with all its deps satisfied.
> 
> Half a minute if you disable backtracking which you don't need. :)

Or if you don't have a supercharged turbo-CPU ... no joke ...

Half an hour.

Yeah, on an E-350 it's GREAT FUN to WAIT for portage ;)

It took me three days of frustration just to figure out an upgrade path.

Re: [gentoo-dev] Re: RFC: storing predefined INSTALL_MASK directory lists in repos

[Donnie Berkholz]

On 19:59 Sat 11 Jan , Peter Stuge wrote:
> Duncan wrote:
> > >> Michał Górny wrote:
> > >>   INSTALL_MASK="systemd bash-completion"
> > >> 
> > >> What are your thoughts?
> > > 
> > > It seems like this will generally duplicate all -USE flags.
> > > 
> > > Would it make sense to instead have a single setting which changes 
> > > the meaning of USE to be that everything not USEd is 
> > > install-masked?
> > 
> > No, this would not be a duplicate.
> 
> I did generalize, but think more about this - certainly for both 
> Michał's examples I have already either set or unset "systemd" and 
> "bash-completion" in USE.

It would largely be a duplicate, since we've already made a decision 
about whether the ability to build without a feature is important enough 
to merit a USE flag. This is essentially rethinking the same decision 
and adding complexity to it. I think having this as an additional 
feature in the core PM would be confusing to users.

It would probably be a better fit in gentoolkit or a similar tool.

-- 
Thanks,
Donnie

Donnie Berkholz
Council Member / Sr. Developer, Gentoo Linux 
Analyst, RedMonk 


pgpcJ3C4i8c3x.pgp
Description: PGP signature

Re: [gentoo-dev] Default USE changes for fortran and mudflap?

[Donnie Berkholz]

On 01:53 Sun 12 Jan , Ryan Hill wrote:
> fortran:
> Do we want to keep enabling fortran by default?  The majority of users 
> will never get the urge to install a fortran package, and the fortran 
> eclass handles those that do.  I think it should be treated as all the 
> other optional languages and disabled by default, but I'd like to know 
> if there are other opinions.

It's essentially only used by the small minority of people installing 
sci-* packages so I'd be fine with that. Reasonable defaults FTW.

-- 
Thanks,
Donnie

Donnie Berkholz
Council Member / Sr. Developer, Gentoo Linux 
Analyst, RedMonk 


pgpKSLofMizut.pgp
Description: PGP signature

Re: [gentoo-dev] Re: [OT] pkgcore bikeshed

[Donnie Berkholz]

On 11:02 Mon 13 Jan , Steven J. Long wrote:
> On Mon, Jan 13, 2014 at 04:15:37PM +0700, "C. Bergström" wrote:
> > > Realistically, we have to keep updating them both in parallel. 
> > > pkgcore needs to be brought up to portage-level functionality,
> 
> Yeah but it already outshines under the hood: all you're talking about 
> is EAPI and radhermit is working on it; I'm sure he and dol-sen would 
> be happy for more help as well, so long as it's supportive. ISTR 
> TomWij is involved too.

Working on the UX of the emerge frontend is also of extreme importance 
if you want people actually using it. Speed is absolutely a usability 
feature but it's not the only one that matters. Maintaining EAPI parity 
is table stakes.

-- 
Thanks,
Donnie

Donnie Berkholz
Council Member / Sr. Developer, Gentoo Linux 
Analyst, RedMonk 


pgpQJ5K5w1XJT.pgp
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Ciaran McCreesh]

On Mon, 13 Jan 2014 19:27:36 +0100
Tom Wijsman  wrote:
> > Not an API. APIs are bad. What we should have is a good set of
> > lightweight Unix-friendly command line tools. See, for example, the
> > "Scripting Commands" section of "man cave".
> 
> It still is an API that way, just expressed differently

The term "API" seems to mean "library or communication interface" these
days (see "Git API" and the like). So for clarity, it's probably better
to make the distinction.

> if you would only do this you're introducing forks where you might
> not need them. Providing shell commands is one possible binding to
> the API...

Forks are far less of a big deal than trying to share a process between
programming languages. This is a *huge* deal when trying to integrate
modern C++ with languages with primitive thread-hostile runtimes like
Python and Ruby.

Unless you're doing a heavy GUI (in which case portability probably
isn't going to cut it anyway, without a far larger API than it's worth
providing), most of the cost for most of the use cases for this is in
the PM reading in configs and the like. The way around that is to allow
communication via pipes, still in a Unix-friendly manner. This can be
implemented nearly transparently.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Tom Wijsman]

On Mon, 13 Jan 2014 18:21:58 +
Ciaran McCreesh  wrote:

> On Mon, 13 Jan 2014 19:16:45 +0100
> Tom Wijsman  wrote:
> > On Mon, 13 Jan 2014 08:49:17 -0800
> > Alec Warner  wrote:
> > > The caching may not be of use, depending on your configuration.
> > > (For example, if you use a gentoo-x86 checkout as your main repo,
> > > you will probably want to run generate cache entries whenever you
> > > cvs up.) It is there to cache ebuild metadata, because if your
> > > depgraph has a few thousand nodes, having to spawn bash to
> > > generate the metadata for every node is very expensive.
> > 
> > It sounds crazy to use gentoo-x86 without a cache; to some extent,
> > the earlier mentions on this ML by Ciaran that we should improve
> > input would make more sense if would want to run without a cache.
> > But I feel like this shouldn't be a topic of discussion unless we
> > intent 'works fast on a plain CV checkout' to be an actual feature.
> 
> I think you're misunderstanding that comment. It's not just the file
> formats that are bad. It's also what data is provided. Two examples
> are the massive lengths we have to go to to figure out what to do
> with || dependencies (thanks to the horrific abuses commonly used in
> the tree...), and the well-documented REQUIRED_USE screwup.

Iotw, 'input'; but thank you for clarifying.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Tom Wijsman]

On Mon, 13 Jan 2014 18:07:39 +
Ciaran McCreesh  wrote:

> On Mon, 13 Jan 2014 15:46:59 +0100
> Tom Wijsman  wrote:
> 
> > On Mon, 13 Jan 2014 16:15:37 +0700
> > "C. Bergström"  wrote:
> > 
> > > Long term the API to pkgcore could be beneficial, but 
> > > again I'm not sure it's a game changer for users.
> > 
> > Long term, we should have an independent API backend that tools can
> > query; not rewrite our tools every time users want to use them with
> > a different package manager. 
> 
> Not an API. APIs are bad. What we should have is a good set of
> lightweight Unix-friendly command line tools. See, for example, the
> "Scripting Commands" section of "man cave".

It still is an API that way, just expressed differently; if you
would only do this you're introducing forks where you might not need
them. Providing shell commands is one possible binding to the API...

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Ciaran McCreesh]

On Mon, 13 Jan 2014 19:16:45 +0100
Tom Wijsman  wrote:
> On Mon, 13 Jan 2014 08:49:17 -0800
> Alec Warner  wrote:
> > The caching may not be of use, depending on your configuration. (For
> > example, if you use a gentoo-x86 checkout as your main repo, you
> > will probably want to run generate cache entries whenever you cvs
> > up.) It is there to cache ebuild metadata, because if your depgraph
> > has a few thousand nodes, having to spawn bash to generate the
> > metadata for every node is very expensive.
> 
> It sounds crazy to use gentoo-x86 without a cache; to some extent, the
> earlier mentions on this ML by Ciaran that we should improve input
> would make more sense if would want to run without a cache. But I feel
> like this shouldn't be a topic of discussion unless we intent 'works
> fast on a plain CV checkout' to be an actual feature.

I think you're misunderstanding that comment. It's not just the file
formats that are bad. It's also what data is provided. Two examples are
the massive lengths we have to go to to figure out what to do with ||
dependencies (thanks to the horrific abuses commonly used in the
tree...), and the well-documented REQUIRED_USE screwup.


-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Tom Wijsman]

On Mon, 13 Jan 2014 18:05:21 +
Ciaran McCreesh  wrote:

> On Mon, 13 Jan 2014 16:46:08 +0100
> Tom Wijsman  wrote:
> > Rebuilds don't cause a different solution in the graph afaik; so, I
> > wouldn't see how that would form a big problem. I also think this
> > would still be covered by preserved-rebuild and/or revdep-rebuild
> > afterwards.
> 
> There used to be a "feature" whereby if resolution took too long,
> you'd get an incomplete answer. The second time you ran the
> resolution, the time could differ, so you'd get a different
> incomplete answer. Dunno if this has been fixed yet.

We should document what Portage does; but from the word 'backtrack' I
think it tries with what it is given, and if it doesn't succeed it goes
one step back and tries to do it with more. But how exactly that is
implemented, I think only people like Zac and Arfrever can tell us; or
we read through the relevant code ourselves.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Tom Wijsman]

On Mon, 13 Jan 2014 08:49:17 -0800
Alec Warner  wrote:

> The caching may not be of use, depending on your configuration. (For
> example, if you use a gentoo-x86 checkout as your main repo, you will
> probably want to run generate cache entries whenever you cvs up.) It
> is there to cache ebuild metadata, because if your depgraph has a few
> thousand nodes, having to spawn bash to generate the metadata for
> every node is very expensive.

It sounds crazy to use gentoo-x86 without a cache; to some extent, the
earlier mentions on this ML by Ciaran that we should improve input
would make more sense if would want to run without a cache. But I feel
like this shouldn't be a topic of discussion unless we intent 'works
fast on a plain CV checkout' to be an actual feature.

I haven't checked, but doesn't Portage in that case build its own cache?

> It is helpful for users cause it can automatically find solutions for
> users that are otherwise unsolvable (and thus avoids the user having
> to find a solution to the depgraph manually.)

If we document how to process --depgraph=0 output, I believe it could
just as well be used; but yes, it basically involves having to run a
few manual upgrades (emerge -1 ...) and/or masks to get going. But you
can do those things in seconds rather than to wait a few minutes...

Some of these Portage could even do automatically regradless, like for
instance the 'no parents' slot conflicts; in which case I think you can
just always upgrade to the newer version, instead of outputting them it
put them as upgrades at the end of the emerge list I think. But I
haven't investigated that further, I'm just getting tired of seeing 'em.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Tom Wijsman]

On Mon, 13 Jan 2014 18:03:31 +0100
Luis Ressel  wrote:

> No, the problem wasn't that rebuilds weren't done (btw: this is not
> about @preserved-rebuilds, but about subslot dependencies), but that
> updates which would trigger such rebuilds are silently ignored. This
> happened to me yesterday while trying --backtrack=0. The available
> update to dev-haskell/parsec simply didn't show up (haskell ebuilds
> make heavy use of subslot deps), I only noticed this because I knew
> there was in fact an update available (thanks to eix-diff). Only after
> enabling backtracking Portage found the update.
> 
> This might well be a bug, perhaps I'll examine the situation when I've
> got more time. 

That description is broad, it sounds like you didn't pass -uD to emerge;
if it is part of the dependency graph (it should, otherwise this
discussion would be useless) and is an update (it is, due to eix-diff),
then with -uD it would pull that in.

This even doesn't have to do anything with sub-slot dependencies as far
as I know; because why would it check the sub-slot dependencies if it
were an upgrade, the upgrade takes precedence over the sub-slot
dependencies. The other option is that I am confused about this; as you
can see, given the possible scenarios that could happen I might have a
different one on mind than you do.

If backtracking doesn't get to it, it would have to list it as a
conflict; if it doesn't do that at the moment, then we certainly have a
bug somewhere I think. Thank you in advance for investigating this.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Ciaran McCreesh]

On Mon, 13 Jan 2014 15:46:59 +0100
Tom Wijsman  wrote:

> On Mon, 13 Jan 2014 16:15:37 +0700
> "C. Bergström"  wrote:
> 
> > Long term the API to pkgcore could be beneficial, but 
> > again I'm not sure it's a game changer for users.
> 
> Long term, we should have an independent API backend that tools can
> query; not rewrite our tools every time users want to use them with a
> different package manager. 

Not an API. APIs are bad. What we should have is a good set of
lightweight Unix-friendly command line tools. See, for example, the
"Scripting Commands" section of "man cave".

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Brian Dolbec]

On Mon, 2014-01-13 at 15:46 +0100, Tom Wijsman wrote:
> On Mon, 13 Jan 2014 16:15:37 +0700
> "C. Bergström"  wrote:
> 
> > Long term the API to pkgcore could be beneficial, but 
> > again I'm not sure it's a game changer for users.
> 
> Long term, we should have an independent API backend that tools can
> query; not rewrite our tools every time users want to use them with a
> different package manager. 
> 

I have been working towards that for years, but, things keep getting in
the way.  By things, I mean other project work that needs to be done as
well.  I got started in all this working on porthole, which uses
portage's api for much of it's information.  To get more features,
information, I wanted to be able to incorporate some of gentoolkit's
info.  Namely equery, but most of it's working code was embedded with
it's output.  I worked hard at separating out it's working code from
it's output so it would be usable by other tools.  I have also been
working on making a pkgcore backend for porthole.  In doing that it
required making a different backend for portage to get some uniformity
for the frontend.  I have had to put development of those on hold, due
to taking over layman's development.  I gave it a new high level api,
modified it's mid level code and gave it a nice api that can be used by
other apps, guis, etc..  One of the things that came up with layman was
to be able to gpg enable it to verify it's repositories.xml list it
downloads.  I did that.  In so doing created  dev-python/pyGPG a python
lib, which has now brought me in to developing Gentoo-keys (another
project that could use help) to manage gentoo's gpg release, developer
keys, and verification.  Also at the same time a year ago, there was a
lot of talk about moving the default location of the gentoo tree, but it
could not be done with current catalyst code.  So offered to help out as
that project was severely lacking people with decent python skills.
That code base is like what portage code was 8 years ago.  And if you
thought todays portage code is bad... you would cringe to see it's code
from 8 years ago.

Now portage was in trouble, while there were some people stepping in to
fix things, I stepped up to help drive out a new release.  I am now
interim lead till we hold an election.  So most of my time now is spent
steering projects, more than coding.  Hey, it's all work that has to be
done.  So I'm putting out fires where it needs it most.

With that aside.  One of the biggest hurdles new developers face with
pkgcore is the lack of decent api docs and flow charts.  Brian was
brilliant and OCD about attaining speed, but at the cost of difficulty
in following the code and creating the steep learning curve.  I have
been trying to get these other projects to a point I could create the
docs and charts needed so that it would be easier for new developers to
find their way in pkgcores code.  That is when pkgcore will make more
headway at becoming portage's replacement.  But some new fires keep
popping up.


Long story in a nutshell, gentoo could use more GOOD firefighters.

Sorry for the long speech ;)
your friendly gentoo python firefighter

-- 
Brian Dolbec 


signature.asc
Description: This is a digitally signed message part

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Ciaran McCreesh]

On Mon, 13 Jan 2014 16:46:08 +0100
Tom Wijsman  wrote:
> Rebuilds don't cause a different solution in the graph afaik; so, I
> wouldn't see how that would form a big problem. I also think this
> would still be covered by preserved-rebuild and/or revdep-rebuild
> afterwards.

There used to be a "feature" whereby if resolution took too long, you'd
get an incomplete answer. The second time you ran the resolution, the
time could differ, so you'd get a different incomplete answer. Dunno if
this has been fixed yet.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Greg KH]

On Tue, Jan 14, 2014 at 12:42:00AM +0700, "C. Bergström" wrote:
> On 01/14/14 12:37 AM, Greg KH wrote:
> >On Mon, Jan 13, 2014 at 04:15:37PM +0700, "C. Bergström" wrote:
> >>At the end of the day we have one codebase which is "engineered" and
> >>another which has "evolved".
> >I'll take an "evolved" codebase over "engineered" anyday.
> >
> >You do realize that is exactly why Linux has succeeded, right?  The
> >kernel has evolved, and was never "engineered".  There's lots people
> >should be learning from biology...
> >
> >So you are using the benifits of evolution right now on your system,
> >don't knock it, it's proven to work.
> I'll bite - While I don't think nature stopped to properly design interfaces
> along the way. I bet you Linus wouldn't agree with your comment very much.

I don't think you have been paying attention much, I'm directly quoting
Linus:
"Linux is evolution, not intelligent design" - Linus

There are many more statements exactly like this from him over the
years, do a bit of research to dig them up.

> 1) I expect quite a bit of time has gone into (Solaris and Linux) kernel
> interfaces

Time doesn't mean they haven't evolved.

> 2) Any larger or invasive changes require quite a bit of planning, review
> and testing. (Possibly with tests (public/private) to cover a large amount
> of the new/existing feature

And we always get it wrong, so they evolve into something that later on
works.

Seriously, this is how the very system you are using has been created,
it's a well-documented fact (look at our changelogs for details.)

sorry,

greg k-h

Re: [gentoo-dev] [OT] pkgcore bikeshed

[Ulrich Mueller]

> On Mon, 13 Jan 2014, Tom Wijsman wrote:

>> I've started collecting things already some months ago:
>> http://wiki.gentoo.org/wiki/Future_EAPI/EAPI_6_tentative_features

> Just for reference: https://bugs.gentoo.org/show_bug.cgi?id=174380

> According to [1] there are besides the tracker a 78 bugs open, of
> which the Gentoo Wiki page has tentatively selected a 16 bugs (till
> Sep 2013); which means that there are a 52 bugs unconsidered so far.

I'm well aware of this. The features listed at the wiki page are the
ones that a) have been discussed, e.g. in the -dev ML, and where (by
my subjective judgement) there is consensus that they belong in the
next EAPI, b) where we have at least a rough idea how the
specification should look like, and c) that can be implemented in
finite time.

Feel free to lobby for any of the 52 additional features if you
believe that it is missing. (Preferably, patches for PMS and for
Portage should be provided. ;-)

> I'd think that looking at that either EAPI 6 will be big, or
> otherwise an EAPI 7 would follow in the very near future...

>  [1]: https://bugs.gentoo.org/buglist.cgi?quicksearch=future%20eapi

Ulrich


pgprYfjlFROdg.pgp
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Greg KH]

On Mon, Jan 13, 2014 at 04:15:37PM +0700, "C. Bergström" wrote:
> At the end of the day we have one codebase which is "engineered" and
> another which has "evolved".

I'll take an "evolved" codebase over "engineered" anyday.

You do realize that is exactly why Linux has succeeded, right?  The
kernel has evolved, and was never "engineered".  There's lots people
should be learning from biology...

So you are using the benifits of evolution right now on your system,
don't knock it, it's proven to work.

greg k-h

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[C. Bergström]

On 01/14/14 12:37 AM, Greg KH wrote:

On Mon, Jan 13, 2014 at 04:15:37PM +0700, "C. Bergström" wrote:

At the end of the day we have one codebase which is "engineered" and
another which has "evolved".

I'll take an "evolved" codebase over "engineered" anyday.

You do realize that is exactly why Linux has succeeded, right?  The
kernel has evolved, and was never "engineered".  There's lots people
should be learning from biology...

So you are using the benifits of evolution right now on your system,
don't knock it, it's proven to work.
I'll bite - While I don't think nature stopped to properly design 
interfaces along the way. I bet you Linus wouldn't agree with your 
comment very much.


1) I expect quite a bit of time has gone into (Solaris and Linux) kernel 
interfaces
2) Any larger or invasive changes require quite a bit of planning, 
review and testing. (Possibly with tests (public/private) to cover a 
large amount of the new/existing feature


To clarify - by engineering I mean significant forethought into the 
design and implementation. I'm not trying to take away the achievements 
of any particular piece of software. In this case it just one of those 
"you know it when you see it". If I wasn't clear or you don't understand 
what I meant - that's ok because I meant no offense to anyone.

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Fabio Erculiani]

On Mon, Jan 13, 2014 at 5:49 PM, Alec Warner  wrote:
[...]
>
> This is not meant to imply that portage is always fast; there are plenty of
> other modules (such as the aforementioned backtracking) that can take a long
> time to find solutions. That is partially why you see Tomwij turning off
> that feature. It is helpful for users cause it can automatically find
> solutions for users that are otherwise unsolvable (and thus avoids the user
> having to find a solution to the depgraph manually.)

Yeah, correct. But it would be nice if Portage backtrack_depgraph()
would memoize (asynchronously serializing to disk, for instance)
partial results for later use.
AFAIR, last time I checked, it wasn't doing that.

Also, it would be nice if the aux_db cache of the vdb could be stored
in a sqlite3 db rather than in RAM. This dict is consuming a huge cut
of memory for little reason (a single vdb.dbapi.match() can eat 100MB
of RAM).
We know how Python deals with freed memory... Sigh...

>
> -A
>
>>
>>
>> --
>> Luis Ressel 
>> GPG fpr: F08D 2AF6 655E 25DE 52BC  E53D 08F5 7F90 3029 B5BD
>
>



-- 
Fabio Erculiani

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Luis Ressel]

On Mon, 13 Jan 2014 16:46:08 +0100
Tom Wijsman  wrote:

> On Mon, 13 Jan 2014 16:38:59 +0100
> Luis Ressel  wrote:
> 
> > On Mon, 13 Jan 2014 15:58:13 +0100
> > Tom Wijsman  wrote:
> > 
> > > Half a minute if you disable backtracking which you don't need. :)
> > 
> > Which sadly also means that some updates get skipped silently.
> > (Those which would trigger rebuilds of other packages because of
> > sub-slot deps, had that case yesterday).
> 
> Can you give an example of that?
> 
> Rebuilds don't cause a different solution in the graph afaik; so, I
> wouldn't see how that would form a big problem. I also think this
> would still be covered by preserved-rebuild and/or revdep-rebuild
> afterwards.

No, the problem wasn't that rebuilds weren't done (btw: this is not
about @preserved-rebuilds, but about subslot dependencies), but that
updates which would trigger such rebuilds are silently ignored. This
happened to me yesterday while trying --backtrack=0. The available
update to dev-haskell/parsec simply didn't show up (haskell ebuilds
make heavy use of subslot deps), I only noticed this because I knew
there was in fact an update available (thanks to eix-diff). Only after
enabling backtracking Portage found the update.

This might well be a bug, perhaps I'll examine the situation when I've
got more time.


-- 
Luis Ressel 
GPG fpr: F08D 2AF6 655E 25DE 52BC  E53D 08F5 7F90 3029 B5BD


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Alec Warner]

On Mon, Jan 13, 2014 at 7:38 AM, Luis Ressel  wrote:

> On Mon, 13 Jan 2014 15:58:13 +0100
> Tom Wijsman  wrote:
>
> > On Mon, 13 Jan 2014 10:31:56 +0100
> > Fabio Erculiani  wrote:
> >
> > > Portage can still take *minutes* to calculate the merge queue of a
> > > pkg with all its deps satisfied.
> >
> > Half a minute if you disable backtracking which you don't need. :)
>
> Which sadly also means that some updates get skipped silently. (Those
> which would trigger rebuilds of other packages because of sub-slot
> deps, had that case yesterday).
>
> > > Ironically, launching the same emerge command twice, will take more
> > > or less the same time.
> >
> > Determinism results in more or less the same time, that's correct;
> > proper benchmarks would show you a similar result.
>
> I guess he means that the (according to the file sizes) extensive
> caching doesn't seem to be of much use.
>

The caching may not be of use, depending on your configuration. (For
example, if you use a gentoo-x86 checkout as your main repo, you will
probably want to run generate cache entries whenever you cvs up.) It is
there to cache ebuild metadata, because if your depgraph has a few thousand
nodes, having to spawn bash to generate the metadata for every node is very
expensive. That being said, for most users that use rsync, the metadata is
pre-generated. As long as they are not changing the cache indicators (not
sure if this is mtime or md5sum these days) they should be seeing a benefit.

This is not meant to imply that portage is always fast; there are plenty of
other modules (such as the aforementioned backtracking) that can take a
long time to find solutions. That is partially why you see Tomwij turning
off that feature. It is helpful for users cause it can automatically find
solutions for users that are otherwise unsolvable (and thus avoids the user
having to find a solution to the depgraph manually.)

-A


>
> --
> Luis Ressel 
> GPG fpr: F08D 2AF6 655E 25DE 52BC  E53D 08F5 7F90 3029 B5BD
>

Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead

[Alec Warner]

On Mon, Jan 13, 2014 at 7:38 AM, Tom Wijsman  wrote:

> On Mon, 13 Jan 2014 15:46:12 +0100
> Peter Stuge  wrote:
>
> > Tom Wijsman wrote:
> > > So, yes, we need more people on pkgcore; no, we can't just leave
> > > Portage behind, as it still is the beating heart of Gentoo for now.
> >
> > I guess the point is that it might continue beating long enough mostly
> > on its own.
>
> How certain are you of this guess? How long is long enough? Mostly?
>
> At the moment, it beats mostly on our efforts to it; and in the near
> future, it'll need to adapt further to newer EAPIs as well as further
> bugs and other complexities in the Portage trees. Otherwise you'll get
> a lot of "we can't continue because they stopped developing Portage".
>
> Hence that's why it comes up every now and then as a subject, hence
> someone finally made the call to reinforcement, ...; so, it won't have
> to continue beating on its own at all.
>
>
As usual, it is nigh impossible to force people to work on a specific
project. I would encourage radhermit to ask for assistance if he thinks he
needs it. dol-sen asked because he is working on a bunch of projects and
knows he cannot maintain portage by himself. Even if the council were to
decree that portage be frozen, I doubt that would actually halt
contributions.

-A



> --
> With kind regards,
>
> Tom Wijsman (TomWij)
> Gentoo Developer
>
> E-mail address  : tom...@gentoo.org
> GPG Public Key  : 6D34E57D
> GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D
>

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Tom Wijsman]

On Mon, 13 Jan 2014 16:38:59 +0100
Luis Ressel  wrote:

> On Mon, 13 Jan 2014 15:58:13 +0100
> Tom Wijsman  wrote:
> 
> > Half a minute if you disable backtracking which you don't need. :)
> 
> Which sadly also means that some updates get skipped silently. (Those
> which would trigger rebuilds of other packages because of sub-slot
> deps, had that case yesterday).

Can you give an example of that?

Rebuilds don't cause a different solution in the graph afaik; so, I
wouldn't see how that would form a big problem. I also think this would
still be covered by preserved-rebuild and/or revdep-rebuild afterwards.

In any case, I've not been experiencing problems with this; but am
interesting to know how this could go wrong, as no backtracking works
for me and I hope it will continue to do so.

Btw, this also makes me question how proper pkgcore can do rebuilds.

> > > Ironically, launching the same emerge command twice, will take
> > > more or less the same time.
> > 
> > Determinism results in more or less the same time, that's correct;
> > proper benchmarks would show you a similar result.
> 
> I guess he means that the (according to the file sizes) extensive
> caching doesn't seem to be of much use.

For that, _all_ caching would need to be removed before the first run;
when there is a mention of "the same", I doubt this was done at all.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Luis Ressel]

On Mon, 13 Jan 2014 15:58:13 +0100
Tom Wijsman  wrote:

> On Mon, 13 Jan 2014 10:31:56 +0100
> Fabio Erculiani  wrote:
> 
> > Portage can still take *minutes* to calculate the merge queue of a
> > pkg with all its deps satisfied.
> 
> Half a minute if you disable backtracking which you don't need. :)

Which sadly also means that some updates get skipped silently. (Those
which would trigger rebuilds of other packages because of sub-slot
deps, had that case yesterday).

> > Ironically, launching the same emerge command twice, will take more
> > or less the same time.
> 
> Determinism results in more or less the same time, that's correct;
> proper benchmarks would show you a similar result.

I guess he means that the (according to the file sizes) extensive
caching doesn't seem to be of much use.


-- 
Luis Ressel 
GPG fpr: F08D 2AF6 655E 25DE 52BC  E53D 08F5 7F90 3029 B5BD


signature.asc
Description: PGP signature

Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead

[Tom Wijsman]

On Mon, 13 Jan 2014 15:46:12 +0100
Peter Stuge  wrote:

> Tom Wijsman wrote:
> > So, yes, we need more people on pkgcore; no, we can't just leave
> > Portage behind, as it still is the beating heart of Gentoo for now.
> 
> I guess the point is that it might continue beating long enough mostly
> on its own.

How certain are you of this guess? How long is long enough? Mostly?

At the moment, it beats mostly on our efforts to it; and in the near
future, it'll need to adapt further to newer EAPIs as well as further
bugs and other complexities in the Portage trees. Otherwise you'll get
a lot of "we can't continue because they stopped developing Portage".

Hence that's why it comes up every now and then as a subject, hence
someone finally made the call to reinforcement, ...; so, it won't have
to continue beating on its own at all.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Tom Wijsman]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mon, 13 Jan 2014 09:56:13 -0500
Ian Stakenvicius  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> On 13/01/14 09:46 AM, Tom Wijsman wrote:
> > On Mon, 13 Jan 2014 16:15:37 +0700 "C. Bergström"
> >  wrote:
> > 
> >> Long term the API to pkgcore could be beneficial, but again I'm
> >> not sure it's a game changer for users.
> > 
> > Long term, we should have an independent API backend that tools
> > can query; not rewrite our tools every time users want to use them
> > with a different package manager.
> 
> ..you mean, define the API and include it in PMS?  how much
> bureaucracy will that need to occur?

The API can be separate of the PMS. The bureaucracy needed for an API
to provide to tools, that doesn't need as much involvement as PMS.

- -- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJS1AbYAAoJEJWyH81tNOV90DEIAIk+zQe4vgJ2XWchQLET064Q
mij/sQQRcI52PZwXFfRh23G+dKkywsPIg+US5KmbWJTtT4UbbLFnn4KCRB/QZE6E
43wHYsKGMgTovpkXQZEy8O4GbFUOb1k9DrsR05r91+QUUUmxxxi9ODq86kavtPVs
UaPyz3me4j3olHLSxzUtxJd5/AgHGwkH+YHtcWny2oaXNUIFpusIPRURqyFyE4xI
Leq/w8/nsmeCc25pPN4cbY6bBwOwCxsZZwik3gmz6FzrAVwCjuRvX8n7szl8viNr
Kd56odFjX/n7pAL+Rj52spz6TNcSe1H8IMkwI9rMYc/LYQSw+/uCGnAXeIxrzWY=
=F2d3
-END PGP SIGNATURE-

Re: [gentoo-dev] Re: [OT] pkgcore bikeshed

[Tom Wijsman]

On Mon, 13 Jan 2014 14:50:44 +0100
Ulrich Mueller  wrote:

> > On Mon, 13 Jan 2014, Andreas K Huettel wrote:
> 
> > So far, I dont know of any work on the exact EAPI-6 feature set yet.
> > We should maybe open a new thread on that, *if* there is already
> > something.
> 
> I've started collecting things already some months ago:
> http://wiki.gentoo.org/wiki/Future_EAPI/EAPI_6_tentative_features

Just for reference: https://bugs.gentoo.org/show_bug.cgi?id=174380

According to [1] there are besides the tracker a 78 bugs open, of which
the Gentoo Wiki page has tentatively selected a 16 bugs (till Sep 2013);
which means that there are a 52 bugs unconsidered so far. I'd think that
looking at that either EAPI 6 will be big, or otherwise an EAPI 7 would
follow in the very near future...

 [1]: https://bugs.gentoo.org/buglist.cgi?quicksearch=future%20eapi

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] Re: [OT] pkgcore bikeshed

[Tom Wijsman]

On Mon, 13 Jan 2014 11:02:10 +
"Steven J. Long"  wrote:

> Yeah but it already outshines under the hood: all you're talking
> about is EAPI and radhermit is working on it;

pkgcore's response to EAPI 6 is something to hold your breath for.

> I'm sure he and dol-sen would be happy for more help as well, so long
> as it's supportive. ISTR TomWij is involved too.

Yes and no, in the short run I'm going to do some repoman work as to
benefit both the QA and Portage teams (and the community as a whole);
from that point on I'll try to look at Portage and pkgcore from a broad
view (software re-engineering style) to see how I want to continue.

As stated in other mails, I'll definitely help keep Portage alive for
as long as possible regardless; I will thus have to see if remaining
time permits to help out with pkgcore as well.

So, in the short run; no, also, I'm devaway for the next two weeks.

Like Portage made out a call for people, I think pkgcore should in the
near future make out a call for more people; there definitely are some
people here and there that have deep interest in it, thus contributions
to it might just be waiting around the corner...

> Updating both in parallel isn't hard: once pkgcore is up to EAPI-5,
> EAPI-6 isn't that much work (mostly bash afair.)

Its feature set is unknown afaik, thus it could become easy but just
as well become huge; this depends on the approach that the PMS team
takes going forward. But for all I know I've seen some things get
implemented in Portage, is the same true for pkgcore?

> At that point, put portage into feature-freeze, and only bugfix it.

Impossible with a new EAPI around the corner.

> Call a hiatus on new EAPIs for 6 months and put all effort into
> making damn sure pkgcore is a drop-in replacement.

This stops the progress of part of the distribution; actually, we've
stopped progress on that for quite some time already as we're already
running a bit behind on the yearly release of a new EAPI.

> > > We could have a technical
> > > discussion on the merits of pkgcore vs. portage, but in the end,
> > > it's about the users.
> 
> I don't think anyone who's serious can come down on any side but
> pkgcore in that debate, so I agree we skip that discussion.

I could indeed conclude the opposite, evening it out; it's pointless to
discuss that right now. We're currently discussing the PMs' futures;
but we're not discussing "the new package manager default" yet, neither
are we discussing a well planned proper migration yet at this time.

> > We can blah blah about performance of resolving package dependencies
> > all day long, but it's clearly not a game changer feature for users.
> > (or they just don't know)
> 
> They just don't know: and it really is a game-changer, once you've
> tried it. We must be able to finally deliver pkgcore speed, 5 years
> after the event.

With Portage only spending half a minute* a day on it here, /care; and
as seen in benchmarks, Portage allows quite some improvements to it.
So, if needed, it is certainly possible to bring its time down.

 * --backtrack=0 is my new favorite parameter, don't need more. :)

> > [1] /* Side details */
> > In a nutshell we make it possible to track the results of "make
> > check" or equivalent test coverage for every source package.
> > There's a little work involved for setting up each package, but it
> > gives the easy ability to *know* and prove that between xyz
> > ${FLAGS} there's no regressions. For example: How do you *know*
> > that fftw or ssl is regression free when you enable avx, fma or
> > some other higher level of optimization (-O3). By knowing I don't
> > mean just result correctness, but also potentially runtime
> > performance, code size and potentially compile times. (If those are
> > things you care about)
> 
> OFC they are, or we wouldn't be using Gentoo ;) And that does sound
> like an interesting project, of wider interest to the community.
> (hint hint;) 

Now that I read this again:

Performance, size and compile times mean nothing if you don't get a
correct result; it all gets funny when the scrollbar of your browser
breaks just because one of its many deep dependencies was built with
-Ofast, then that small bit of extra performance can be laughed at.

 And yeah, searching that dependency is also a lot of fun; which
means you'll want to recompile libraries with -O2 again until you have
found it, unless you want to spend days figuring it out. And for this
last thing, it's handy you can grep flags in the /var/db/pkg/. 

 At which point you can be lucky it isn't done by some package
that has bad QA and has overridden CFLAGS or something like that. 

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Tom Wijsman]

On Mon, 13 Jan 2014 10:31:56 +0100
Fabio Erculiani  wrote:

> Portage can still take *minutes* to calculate the merge queue of a
> pkg with all its deps satisfied.

Half a minute if you disable backtracking which you don't need. :)

> Ironically, launching the same emerge command twice, will take more
> or less the same time.

Determinism results in more or less the same time, that's correct;
proper benchmarks would show you a similar result.

> Yeah, this is probably bad design...

Too broad generalization with no fundamental argument about the design.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Ian Stakenvicius]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 13/01/14 09:46 AM, Tom Wijsman wrote:
> On Mon, 13 Jan 2014 16:15:37 +0700 "C. Bergström"
>  wrote:
> 
>> Long term the API to pkgcore could be beneficial, but again I'm
>> not sure it's a game changer for users.
> 
> Long term, we should have an independent API backend that tools
> can query; not rewrite our tools every time users want to use them
> with a different package manager.

..you mean, define the API and include it in PMS?  how much
bureaucracy will that need to occur?


-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlLT/o0ACgkQ2ugaI38ACPBIlwD/UhmM7jiu0c3kbczm/p0eA0nE
QklKzRWkYxLpUpWEwegA/0GzTsFmUL/9C+xVvh1rpKjk23L6LAR/mWuiB34Vmnb0
=vpX0
-END PGP SIGNATURE-

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Tom Wijsman]

On Mon, 13 Jan 2014 16:15:37 +0700
"C. Bergström"  wrote:

> At the end of the day we have one codebase which is
> "engineered" and another which has "evolved".

Too broad generalization, too much assumption; both can be held as
meaning nothing compared to what "engineered" and "evolved" could
really be, but as with doing that, it gets a subjective nature.

In other words, the lack of context makes this statement meaningless.

What we're looking for is a well-engineered _and_ well-evolving
distribution, not just a well-engineered _or_ well-evolving package;
and thus there's a lot more to it to really use this as an argument.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead

[Peter Stuge]

Tom Wijsman wrote:
> So, yes, we need more people on pkgcore; no, we can't just leave
> Portage behind, as it still is the beating heart of Gentoo for now.

I guess the point is that it might continue beating long enough mostly
on its own.


//Peter


pgp_E4oOCUrAm.pgp
Description: PGP signature

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Tom Wijsman]

On Mon, 13 Jan 2014 16:15:37 +0700
"C. Bergström"  wrote:

> Long term the API to pkgcore could be beneficial, but 
> again I'm not sure it's a game changer for users.

Long term, we should have an independent API backend that tools can
query; not rewrite our tools every time users want to use them with a
different package manager. 

Yes, a side effect of not having that is that a lot of tools would need
to be rewritten too if we would make a sudden move to pkgcore; or at
least, it would mean that Portage would still need to be around to serve
as an API, but that would also mean it still needs to be maintained.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead

[Tom Wijsman]

On Mon, 13 Jan 2014 15:39:17 +0700
C. Bergström  wrote:

> Drive-by trolling comment but I wish the effort to keep porkage alive
> would have instead been directed towards pkgcore.

If we still have users left by the time pkgcore is finished...

Moving the work efforts from one PM to another and expecting the future
to go well is too optimistic, of course it would make pkgcore a great
thing eventually but it can come at the cost of Gentoo itself; *danger*.

While Portage's performance might have dropped a little (though is still
quite fast [~30s] on --backtrack=0), the development on it didn't stop
as Arfrever and few_ have been committing over the past weeks to months;
there just hasn't been a release because the lead position was MIA...

And before anyone pulls a "pkgcore almost has EAPI 5 support" comment;
you should note that EAPI 6 is around the corner, and many of its
would-be features have already been implemented in Portage.

I'm in doubt if the same hold true for pkgcore; which was known to be
dead for a while as its founder left development, and it took some time
for two of our developers to pikc it up again.

So, yes, we need more people on pkgcore; no, we can't just leave
Portage behind, as it still is the beating heart of Gentoo for now.

Who knows, as we get up to speed, Portage might even see some neat
refactoring and performance improvements; or well, that's what I expect
to see some of us work towards to. We're already putting measures in
place to clean up the source code (short variable names, file length);
but for now, it's not that huge of a problem to stop using it at all.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : tom...@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D


signature.asc
Description: PGP signature

Re: [gentoo-dev] Re: [OT] pkgcore bikeshed

[Ulrich Mueller]

> On Mon, 13 Jan 2014, Andreas K Huettel wrote:

> So far, I dont know of any work on the exact EAPI-6 feature set yet.
> We should maybe open a new thread on that, *if* there is already
> something.

I've started collecting things already some months ago:
http://wiki.gentoo.org/wiki/Future_EAPI/EAPI_6_tentative_features

Ulrich


pgpzCtBa7Qe2A.pgp
Description: PGP signature

Re: Re: [gentoo-dev] Re: [OT] pkgcore bikeshed

[Andreas K. Huettel]

Am Montag 13 Januar 2014, 13:28:13 schrieb Alexander Berntsen:

> > Updating both in parallel isn't hard: once pkgcore is up to EAPI-5,
> >  EAPI-6 isn't that much work (mostly bash afair.)

So far, I dont know of any work on the exact EAPI-6 feature set yet. We should 
maybe open a new thread on that, *if* there is already something.

-- 
Andreas K. Huettel
Gentoo Linux developer
kde, council

Re: [gentoo-dev] Re: [OT] pkgcore bikeshed

[Alexander Berntsen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 13/01/14 12:02, Steven J. Long wrote:
> Yeah but it already outshines under the hood: all you're talking 
> about is EAPI and radhermit is working on it; I'm sure he and 
> dol-sen would be happy for more help as well, so long as it's 
> supportive. ISTR TomWij is involved too.
I know all this. But until the EAPI is brought up to date, I don't see a
lot of devs migrating their work there. (Kudos to radhermit for his
work.) Furthermore, bringing the EAPI up to date is not of ignorable
complexity.

> Updating both in parallel isn't hard: once pkgcore is up to EAPI-5,
>  EAPI-6 isn't that much work (mostly bash afair.)
If it is trivial: show us the code.

> At that point, put portage into feature-freeze, and only bugfix it.
>  Call a hiatus on new EAPIs for 6 months and put all effort into 
> making damn sure pkgcore is a drop-in replacement.
I don't see this happening as it stands right now. Let's revisit this
when pkgcore is more up to date.

> There's certainly enough devs to do that, and definitely enough 
> interest in finally moving to portage-NG.
These are two big statements, and I'm not convinced either of them
holds true. Feel free to gather data on the latter. (The former is not
immediately quantifiable.)

- -- 
Alexander
alexan...@plaimi.net
http://plaimi.net/~alexander
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlLT290ACgkQRtClrXBQc7XeuAD+Jj4Oj73br8pvcKdkDSVFfE3J
tG6ajS3wzQHNmkbxWPoA+gMVyzgm5SSDRAjqHysocgvvDe6Xwa2CA+/ON98u+2xs
=mBxD
-END PGP SIGNATURE-

[gentoo-dev] Re: [OT] pkgcore bikeshed

[Steven J. Long]

On Mon, Jan 13, 2014 at 04:15:37PM +0700, "C. Bergström" wrote:
> On 01/13/14 03:43 PM, Alexander Berntsen wrote:
> > Realistically, we have to keep updating them both in parallel. pkgcore
> > needs to be brought up to portage-level functionality,

Yeah but it already outshines under the hood: all you're talking about is
EAPI and radhermit is working on it; I'm sure he and dol-sen would be
happy for more help as well, so long as it's supportive. ISTR TomWij is
involved too.

Updating both in parallel isn't hard: once pkgcore is up to EAPI-5, EAPI-6
isn't that much work (mostly bash afair.) At that point, put portage into
feature-freeze, and only bugfix it. Call a hiatus on new EAPIs for 6 months
and put all effort into making damn sure pkgcore is a drop-in replacement.

There's certainly enough devs to do that, and definitely enough interest
in finally moving to portage-NG.

> > and we need to keep fixing portage bugs for its many users.

Sure.

> > We could have a technical
> > discussion on the merits of pkgcore vs. portage, but in the end, it's
> > about the users.

I don't think anyone who's serious can come down on any side but pkgcore
in that debate, so I agree we skip that discussion.

You won't get any disagreement from me on your last point.

> > For the record, I would be very happy if you hacked on pkgcore. Just
> > as happy as if you hacked on portage, even. So please do. :-)

Good good :-)

> We can blah blah about performance of resolving package dependencies
> all day long, but it's clearly not a game changer feature for users.
> (or they just don't know)

They just don't know: and it really is a game-changer, once you've tried it.
We must be able to finally deliver pkgcore speed, 5 years after the event.

>  Long term the API to pkgcore could be beneficial, but again I'm not sure
> it's a game changer for users.

Doesn't matter: it's good to have something the devs can get hyped about too.
snakeoil is a lovely bit of code.

> [1] /* Side details */
> In a nutshell we make it possible to track the results of "make check" 
> or equivalent test coverage for every source package. There's a little 
> work involved for setting up each package, but it gives the easy ability 
> to *know* and prove that between xyz ${FLAGS} there's no regressions. 
> For example: How do you *know* that fftw or ssl is regression free when 
> you enable avx, fma or some other higher level of optimization (-O3). By 
> knowing I don't mean just result correctness, but also potentially 
> runtime performance, code size and potentially compile times. (If those 
> are things you care about)

OFC they are, or we wouldn't be using Gentoo ;) And that does sound like
an interesting project, of wider interest to the community. (hint hint;)
 
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[C. Bergström]

On 01/13/14 04:31 PM, Fabio Erculiani wrote:

On Mon, Jan 13, 2014 at 10:15 AM, "C. Bergström"
 wrote:

On 01/13/14 03:43 PM, Alexander Berntsen wrote:
Where I work uses pkgcore[1], but not the areas which are generally
beneficial to the whole community. (We use it as part of a web application
to handle testsuites which have build dependencies.) We can blah blah about
performance of resolving package dependencies all day long,
[...]

Not sure about what you mean with "blah blah". But given the amount of
both disk caches (metadata, vdb cache) and memory caches (the
in-memory aux_db cache that portage loads using pickle (it's a dict)
takes like 70-100Mb of RAM on an average desktop system), Portage can
still take *minutes* to calculate the merge queue of a pkg with all
its deps satisfied. Ironically, launching the same emerge command
twice, will take more or less the same time.
Yeah, this is probably bad design...
ack - I know the benefits (and downsides) of pkgcore compared to 
portage, but I leave that up to others who would like to voice their 
opinion. It would be great to get pkgcore up to feature parity with 
portage, but I don't have the resources to help with that. (In the 
future, possibly next month, I will try to put some bounties)

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[Fabio Erculiani]

On Mon, Jan 13, 2014 at 10:15 AM, "C. Bergström"
 wrote:
> On 01/13/14 03:43 PM, Alexander Berntsen wrote:
> Where I work uses pkgcore[1], but not the areas which are generally
> beneficial to the whole community. (We use it as part of a web application
> to handle testsuites which have build dependencies.) We can blah blah about
> performance of resolving package dependencies all day long,
> [...]

Not sure about what you mean with "blah blah". But given the amount of
both disk caches (metadata, vdb cache) and memory caches (the
in-memory aux_db cache that portage loads using pickle (it's a dict)
takes like 70-100Mb of RAM on an average desktop system), Portage can
still take *minutes* to calculate the merge queue of a pkg with all
its deps satisfied. Ironically, launching the same emerge command
twice, will take more or less the same time.
Yeah, this is probably bad design...


-- 
Fabio Erculiani

Re: [gentoo-dev] [OT] pkgcore bikeshed (was Portage team)

[C. Bergström]

On 01/13/14 03:43 PM, Alexander Berntsen wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 13/01/14 09:39, C. Bergström wrote:

Drive-by trolling comment but I wish the effort to keep porkage
alive would have instead been directed towards pkgcore.

Realistically, we have to keep updating them both in parallel. pkgcore
needs to be brought up to portage-level functionality, and we need to
keep fixing portage bugs for its many users. We could have a technical
discussion on the merits of pkgcore vs. portage, but in the end, it's
about the users.

For the record, I would be very happy if you hacked on pkgcore. Just
as happy as if you hacked on portage, even. So please do. :-)
Where I work uses pkgcore[1], but not the areas which are generally 
beneficial to the whole community. (We use it as part of a web 
application to handle testsuites which have build dependencies.) We can 
blah blah about performance of resolving package dependencies all day 
long, but it's clearly not a game changer feature for users. (or they 
just don't know)  Long term the API to pkgcore could be beneficial, but 
again I'm not sure it's a game changer for users. Dare I turn this from 
a trolling comment into a laundry list of pros/cons of pkgcore and why 
now may be a good time to invest in the future. At the end of the day we 
have one codebase which is "engineered" and another which has "evolved". 
If gentoo only ever wants to fetch tarballs and build source.. who 
cares.. If you ever want to build something on top of that - you will 
painfully realize that there's only 1 choice.


On 01/13/14 03:59 PM, Dirkjan Ochtman wrote:
If you know your email is going to be drive-by trolling, maybe just 
hold it in next time? 

Yeah I know better, but this time was like a fart - (half attempt joking)



[1] /* Side details */
In a nutshell we make it possible to track the results of "make check" 
or equivalent test coverage for every source package. There's a little 
work involved for setting up each package, but it gives the easy ability 
to *know* and prove that between xyz ${FLAGS} there's no regressions. 
For example: How do you *know* that fftw or ssl is regression free when 
you enable avx, fma or some other higher level of optimization (-O3). By 
knowing I don't mean just result correctness, but also potentially 
runtime performance, code size and potentially compile times. (If those 
are things you care about)

Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead

[Dirkjan Ochtman]

On Mon, Jan 13, 2014 at 9:39 AM, C. Bergström  wrote:
> Drive-by trolling comment but I wish the effort to keep porkage alive would
> have instead been directed towards pkgcore.

If you know your email is going to be drive-by trolling, maybe just
hold it in next time?

Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead

[Alexander Berntsen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 13/01/14 09:39, C. Bergström wrote:
> Drive-by trolling comment but I wish the effort to keep porkage 
> alive would have instead been directed towards pkgcore.
Realistically, we have to keep updating them both in parallel. pkgcore
needs to be brought up to portage-level functionality, and we need to
keep fixing portage bugs for its many users. We could have a technical
discussion on the merits of pkgcore vs. portage, but in the end, it's
about the users.

For the record, I would be very happy if you hacked on pkgcore. Just
as happy as if you hacked on portage, even. So please do. :-)
- -- 
Alexander
alexan...@plaimi.net
http://plaimi.net/~alexander
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlLTpx8ACgkQRtClrXBQc7UXtAEAp+ZlhX1E7NGp4rF0x8yEVzFk
Ynf1W9rr0Q2XZ3nBsk4A/RsrKd82zgYPaBizcs2toz/U9pjmbOkS9PUQPJAFe4ze
=EKI+
-END PGP SIGNATURE-

Re: [gentoo-dev] Portage team, Zac's development break and stepping down as lead

[C . Bergström]

Drive-by trolling comment but I wish the effort to keep porkage alive would have instead been directed towards pkgcore.