Re: [gentoo-portage-dev] [PATCH 1/2] portdbapi: add async_aux_get method (bug 648790)

[Zac Medico]

On 02/26/2018 11:58 AM, Michał Górny wrote:
> W dniu pon, 26.02.2018 o godzinie 11∶40 -0800, użytkownik Zac Medico
> napisał:
>> On 02/26/2018 11:23 AM, Michał Górny wrote:
>>> W dniu pon, 26.02.2018 o godzinie 10∶09 -0800, użytkownik Zac Medico
>>> napisał:
 On 02/26/2018 04:29 AM, Michał Górny wrote:
> W dniu nie, 25.02.2018 o godzinie 17∶50 -0800, użytkownik Zac Medico
> napisał:
>> Add async_aux_get method that returns a Future and otherwise
>> behaves identically to aux_get. Use async_aux_get to implement
>> the synchronous aux_get method.
>>
>> Bug: https://bugs.gentoo.org/648790
>> ---
>>  pym/portage/dbapi/porttree.py | 91 
>> +--
>>  1 file changed, 70 insertions(+), 21 deletions(-)
>>
>
> What's the exact use case? What's the gain, in numbers?

 I'm planning to use the for repoman, since it commonly has to generate
 metadata for multiple packages a once, when dealing with ebuild or
 eclass modifications.

 The gain in numbers is that it scales linearly with the number of
 available cores.

> This seems like a lot of added complexity.

 It doesn't really add complexity, asynchronous code or this sort is
 already used throughout portage internals. Asynchronous methods like
 these are quite standard today, thanks to asyncio.

> With Portage being practically unmaintainable,

 Any reasonably complex codebase appears to be unmaintainable to those
 who don't maintain it.

> I'm against any changes that make things worse without
> clearly defined major benefit.

 Transitioning to asynchronous interfaces like this is unavoidable as we
 move into the realm of asyncio compatibility.
>>>
>>> I think we can agree that functions like aux_get() that rely on callers
>>> providing constant parameters to obtain the necessary output are far
>>> from optimal. I believe Portage should aim towards transitioning callers
>>> to more object-oriented APIs.
>>
>> That's fine but aux_get is a public API that will remain supported until
>> we have a replacement, and async_aux_get will be an immediately useful
>> enhancement to an existing API.
> 
> No, it will become a new public API which we will end up supporting
> forever.

We can deprecate both aux_get and async_aux_get after an alternative API
is available in a stable release of portage.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH 1/2] portdbapi: add async_aux_get method (bug 648790)

[Michał Górny]

W dniu pon, 26.02.2018 o godzinie 11∶40 -0800, użytkownik Zac Medico
napisał:
> On 02/26/2018 11:23 AM, Michał Górny wrote:
> > W dniu pon, 26.02.2018 o godzinie 10∶09 -0800, użytkownik Zac Medico
> > napisał:
> > > On 02/26/2018 04:29 AM, Michał Górny wrote:
> > > > W dniu nie, 25.02.2018 o godzinie 17∶50 -0800, użytkownik Zac Medico
> > > > napisał:
> > > > > Add async_aux_get method that returns a Future and otherwise
> > > > > behaves identically to aux_get. Use async_aux_get to implement
> > > > > the synchronous aux_get method.
> > > > > 
> > > > > Bug: https://bugs.gentoo.org/648790
> > > > > ---
> > > > >  pym/portage/dbapi/porttree.py | 91 
> > > > > +--
> > > > >  1 file changed, 70 insertions(+), 21 deletions(-)
> > > > > 
> > > > 
> > > > What's the exact use case? What's the gain, in numbers?
> > > 
> > > I'm planning to use the for repoman, since it commonly has to generate
> > > metadata for multiple packages a once, when dealing with ebuild or
> > > eclass modifications.
> > > 
> > > The gain in numbers is that it scales linearly with the number of
> > > available cores.
> > > 
> > > > This seems like a lot of added complexity.
> > > 
> > > It doesn't really add complexity, asynchronous code or this sort is
> > > already used throughout portage internals. Asynchronous methods like
> > > these are quite standard today, thanks to asyncio.
> > > 
> > > > With Portage being practically unmaintainable,
> > > 
> > > Any reasonably complex codebase appears to be unmaintainable to those
> > > who don't maintain it.
> > > 
> > > > I'm against any changes that make things worse without
> > > > clearly defined major benefit.
> > > 
> > > Transitioning to asynchronous interfaces like this is unavoidable as we
> > > move into the realm of asyncio compatibility.
> > 
> > I think we can agree that functions like aux_get() that rely on callers
> > providing constant parameters to obtain the necessary output are far
> > from optimal. I believe Portage should aim towards transitioning callers
> > to more object-oriented APIs.
> 
> That's fine but aux_get is a public API that will remain supported until
> we have a replacement, and async_aux_get will be an immediately useful
> enhancement to an existing API.

No, it will become a new public API which we will end up supporting
forever.

> 
> > That's why I'm opposed to *proliferating* bad APIs. Maybe I'm wrong but
> > I don't believe this code would be helpful in transitioning Portage to
> > object-oriented API.
> 
> I'd welcome the addition of new APIs, but that should not block the
> enhancement of existing APIs.

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 1/2] portdbapi: add async_aux_get method (bug 648790)

[Zac Medico]

On 02/26/2018 11:23 AM, Michał Górny wrote:
> W dniu pon, 26.02.2018 o godzinie 10∶09 -0800, użytkownik Zac Medico
> napisał:
>> On 02/26/2018 04:29 AM, Michał Górny wrote:
>>> W dniu nie, 25.02.2018 o godzinie 17∶50 -0800, użytkownik Zac Medico
>>> napisał:
 Add async_aux_get method that returns a Future and otherwise
 behaves identically to aux_get. Use async_aux_get to implement
 the synchronous aux_get method.

 Bug: https://bugs.gentoo.org/648790
 ---
  pym/portage/dbapi/porttree.py | 91 
 +--
  1 file changed, 70 insertions(+), 21 deletions(-)

>>>
>>> What's the exact use case? What's the gain, in numbers?
>>
>> I'm planning to use the for repoman, since it commonly has to generate
>> metadata for multiple packages a once, when dealing with ebuild or
>> eclass modifications.
>>
>> The gain in numbers is that it scales linearly with the number of
>> available cores.
>>
>>> This seems like a lot of added complexity.
>>
>> It doesn't really add complexity, asynchronous code or this sort is
>> already used throughout portage internals. Asynchronous methods like
>> these are quite standard today, thanks to asyncio.
>>
>>> With Portage being practically unmaintainable,
>>
>> Any reasonably complex codebase appears to be unmaintainable to those
>> who don't maintain it.
>>
>>> I'm against any changes that make things worse without
>>> clearly defined major benefit.
>>
>> Transitioning to asynchronous interfaces like this is unavoidable as we
>> move into the realm of asyncio compatibility.
> 
> I think we can agree that functions like aux_get() that rely on callers
> providing constant parameters to obtain the necessary output are far
> from optimal. I believe Portage should aim towards transitioning callers
> to more object-oriented APIs.

That's fine but aux_get is a public API that will remain supported until
we have a replacement, and async_aux_get will be an immediately useful
enhancement to an existing API.

> That's why I'm opposed to *proliferating* bad APIs. Maybe I'm wrong but
> I don't believe this code would be helpful in transitioning Portage to
> object-oriented API.

I'd welcome the addition of new APIs, but that should not block the
enhancement of existing APIs.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH 1/2] portdbapi: add async_aux_get method (bug 648790)

[Michał Górny]

W dniu pon, 26.02.2018 o godzinie 10∶09 -0800, użytkownik Zac Medico
napisał:
> On 02/26/2018 04:29 AM, Michał Górny wrote:
> > W dniu nie, 25.02.2018 o godzinie 17∶50 -0800, użytkownik Zac Medico
> > napisał:
> > > Add async_aux_get method that returns a Future and otherwise
> > > behaves identically to aux_get. Use async_aux_get to implement
> > > the synchronous aux_get method.
> > > 
> > > Bug: https://bugs.gentoo.org/648790
> > > ---
> > >  pym/portage/dbapi/porttree.py | 91 
> > > +--
> > >  1 file changed, 70 insertions(+), 21 deletions(-)
> > > 
> > 
> > What's the exact use case? What's the gain, in numbers?
> 
> I'm planning to use the for repoman, since it commonly has to generate
> metadata for multiple packages a once, when dealing with ebuild or
> eclass modifications.
> 
> The gain in numbers is that it scales linearly with the number of
> available cores.
> 
> > This seems like a lot of added complexity.
> 
> It doesn't really add complexity, asynchronous code or this sort is
> already used throughout portage internals. Asynchronous methods like
> these are quite standard today, thanks to asyncio.
> 
> > With Portage being practically unmaintainable,
> 
> Any reasonably complex codebase appears to be unmaintainable to those
> who don't maintain it.
> 
> > I'm against any changes that make things worse without
> > clearly defined major benefit.
> 
> Transitioning to asynchronous interfaces like this is unavoidable as we
> move into the realm of asyncio compatibility.

I think we can agree that functions like aux_get() that rely on callers
providing constant parameters to obtain the necessary output are far
from optimal. I believe Portage should aim towards transitioning callers
to more object-oriented APIs.

That's why I'm opposed to *proliferating* bad APIs. Maybe I'm wrong but
I don't believe this code would be helpful in transitioning Portage to
object-oriented API.

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 1/2] portdbapi: add async_aux_get method (bug 648790)

[Zac Medico]

On 02/26/2018 10:16 AM, Alec Warner wrote:
> 
> 
> On Mon, Feb 26, 2018 at 1:09 PM, Zac Medico  > wrote:
> 
> On 02/26/2018 04:29 AM, Michał Górny wrote:
> > W dniu nie, 25.02.2018 o godzinie 17∶50 -0800, użytkownik Zac Medico
> > napisał:
> >> Add async_aux_get method that returns a Future and otherwise
> >> behaves identically to aux_get. Use async_aux_get to implement
> >> the synchronous aux_get method.
> >>
> >> Bug: https://bugs.gentoo.org/648790
> >> ---
> >>  pym/portage/dbapi/porttree.py | 91 
> +--
> >>  1 file changed, 70 insertions(+), 21 deletions(-)
> >>
> >
> > What's the exact use case? What's the gain, in numbers?
> 
> I'm planning to use the for repoman, since it commonly has to generate
> metadata for multiple packages a once, when dealing with ebuild or
> eclass modifications.
> 
> The gain in numbers is that it scales linearly with the number of
> available cores.
> 
> 
> Daniel, any data on real world gains from the Funtoo implementation?

There's no doubt that it scales linearly, it uses exactly the same code
as egencache --jobs.

> > This seems like a lot of added complexity.
> 
> It doesn't really add complexity, asynchronous code or this sort is
> already used throughout portage internals. Asynchronous methods like
> these are quite standard today, thanks to asyncio.
> 
> > With Portage being practically unmaintainable,
> 
> Any reasonably complex codebase appears to be unmaintainable to those
> who don't maintain it.
> 
> 
> I mean I'm generally with mgorny, but I don't think this change is that
> complicated.
> 
> The 'hard to maintain' stuff for me comes from:
> 
> 1) really long functions (e.g >100 lines) with poor naming (x, my*
> variables, etc.)
> 2) portions of the codebase with no tests (more than I'd like...)
> 3) The random spawl and layout of some components that just makes it
> annoying to modify specific bits.
> 
> While I agree that these are all a problem (and I've tried at various
> times to tilt at these windmills to no avail)
> I don't think this makes any of those worse.

Agreed.

> > I'm against any changes that make things worse without
> > clearly defined major benefit.
> 
> Transitioning to asynchronous interfaces like this is unavoidable as we
> move into the realm of asyncio compatibility.
> 
> 
> So to me this implies we will be adding even more asyncIO to try to
> improve the performance of portage by utilizing more cores?

Correct, and frankly any other python project that can benefit from
multiprocessing should do the same thing.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH 1/2] portdbapi: add async_aux_get method (bug 648790)

[Alec Warner]

On Mon, Feb 26, 2018 at 1:09 PM, Zac Medico  wrote:

> On 02/26/2018 04:29 AM, Michał Górny wrote:
> > W dniu nie, 25.02.2018 o godzinie 17∶50 -0800, użytkownik Zac Medico
> > napisał:
> >> Add async_aux_get method that returns a Future and otherwise
> >> behaves identically to aux_get. Use async_aux_get to implement
> >> the synchronous aux_get method.
> >>
> >> Bug: https://bugs.gentoo.org/648790
> >> ---
> >>  pym/portage/dbapi/porttree.py | 91 ++
> +++--
> >>  1 file changed, 70 insertions(+), 21 deletions(-)
> >>
> >
> > What's the exact use case? What's the gain, in numbers?
>
> I'm planning to use the for repoman, since it commonly has to generate
> metadata for multiple packages a once, when dealing with ebuild or
> eclass modifications.
>
> The gain in numbers is that it scales linearly with the number of
> available cores.
>

Daniel, any data on real world gains from the Funtoo implementation?


>
> > This seems like a lot of added complexity.
>
> It doesn't really add complexity, asynchronous code or this sort is
> already used throughout portage internals. Asynchronous methods like
> these are quite standard today, thanks to asyncio.
>
> > With Portage being practically unmaintainable,
>
> Any reasonably complex codebase appears to be unmaintainable to those
> who don't maintain it.
>

I mean I'm generally with mgorny, but I don't think this change is that
complicated.

The 'hard to maintain' stuff for me comes from:

1) really long functions (e.g >100 lines) with poor naming (x, my*
variables, etc.)
2) portions of the codebase with no tests (more than I'd like...)
3) The random spawl and layout of some components that just makes it
annoying to modify specific bits.

While I agree that these are all a problem (and I've tried at various times
to tilt at these windmills to no avail)
I don't think this makes any of those worse.


>
> > I'm against any changes that make things worse without
> > clearly defined major benefit.
>
> Transitioning to asynchronous interfaces like this is unavoidable as we
> move into the realm of asyncio compatibility.
>

So to me this implies we will be adding even more asyncIO to try to improve
the performance of portage by utilizing more cores?

-A


> --
> Thanks,
> Zac
>
>

Re: [gentoo-portage-dev] [PATCH 1/2] portdbapi: add async_aux_get method (bug 648790)

[Zac Medico]

On 02/26/2018 04:29 AM, Michał Górny wrote:
> W dniu nie, 25.02.2018 o godzinie 17∶50 -0800, użytkownik Zac Medico
> napisał:
>> Add async_aux_get method that returns a Future and otherwise
>> behaves identically to aux_get. Use async_aux_get to implement
>> the synchronous aux_get method.
>>
>> Bug: https://bugs.gentoo.org/648790
>> ---
>>  pym/portage/dbapi/porttree.py | 91 
>> +--
>>  1 file changed, 70 insertions(+), 21 deletions(-)
>>
> 
> What's the exact use case? What's the gain, in numbers?

I'm planning to use the for repoman, since it commonly has to generate
metadata for multiple packages a once, when dealing with ebuild or
eclass modifications.

The gain in numbers is that it scales linearly with the number of
available cores.

> This seems like a lot of added complexity.

It doesn't really add complexity, asynchronous code or this sort is
already used throughout portage internals. Asynchronous methods like
these are quite standard today, thanks to asyncio.

> With Portage being practically unmaintainable,

Any reasonably complex codebase appears to be unmaintainable to those
who don't maintain it.

> I'm against any changes that make things worse without
> clearly defined major benefit.

Transitioning to asynchronous interfaces like this is unavoidable as we
move into the realm of asyncio compatibility.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH 1/2] portdbapi: add async_aux_get method (bug 648790)

[Michał Górny]

W dniu nie, 25.02.2018 o godzinie 17∶50 -0800, użytkownik Zac Medico
napisał:
> Add async_aux_get method that returns a Future and otherwise
> behaves identically to aux_get. Use async_aux_get to implement
> the synchronous aux_get method.
> 
> Bug: https://bugs.gentoo.org/648790
> ---
>  pym/portage/dbapi/porttree.py | 91 
> +--
>  1 file changed, 70 insertions(+), 21 deletions(-)
> 

What's the exact use case? What's the gain, in numbers? This seems like
a lot of added complexity. With Portage being practically
unmaintainable, I'm against any changes that make things worse without
clearly defined major benefit.

-- 
Best regards,
Michał Górny