Re: [gentoo-portage-dev] [PATCH] dblink: add locks for parallel-install with blockers (bug 576888)

[Zac Medico]

On 03/14/2016 03:26 AM, Alexander Berntsen wrote:
> I can't say much more than "ACK, probably makes sense" really. But
> please test this *a lot* before merging it.

Thanks. I'll test it a lot.

> Regarding the merging of this patch, and th egencache patch that has
> already been released: I thought we agreed that .29 should be *only*
> the repoman merger, and then bug fixes go into a .30 where we try to
> get a stable release with the new repoman. Why was egencache merged
> anyway?

Sorry, I didn't really comprehend the necessity of excluding everything
besides the new repoman changes. The egencache patch has no overlap with
repoman the branch, so I didn't see any harm. I'll hold of on merging
anything more.

> Should we not merge repoman to stable ASAP before doing
> anything else? That would make .29 easier.

That works for me.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] dblink: add locks for parallel-install with blockers (bug 576888)

[Alexander Berntsen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

I can't say much more than "ACK, probably makes sense" really. But
please test this *a lot* before merging it.

Regarding the merging of this patch, and th egencache patch that has
already been released: I thought we agreed that .29 should be *only*
the repoman merger, and then bug fixes go into a .30 where we try to
get a stable release with the new repoman. Why was egencache merged
anyway? Should we not merge repoman to stable ASAP before doing
anything else? That would make .29 easier.
- -- 
Alexander
berna...@gentoo.org
https://secure.plaimi.net/~alexander
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCgAGBQJW5pHOAAoJENQqWdRUGk8BE0QQAMep2RpCTpTaW8N3D0iwxmY4
nOJPtvN+BZzWWFW+j0CIkwn3uinuN/NuxOOvnW7ZMwATHsmToPpt1VCsfXcelzTv
atx/1dPkdWCeeEDFEYpECInV+Lj38Sf1tlDIaWRmmDuHUkf71LojUlpvptFgEW7M
TCs7bnxyqQXxIDl+QGxWvFC3vWG14c59mMG2bPV+gKYpf010tc4tGRKPxKtT8Pu/
0JD/cq0KBdFqCh/lPZ6XruayzTQoEd3hYRn/3MPKdT/EyKhtwfGFs785gMDjNSpc
BPWCYH18Ff/s20Rldn2120QiDzK45D/ty/y4tS+rZoIcxzRycr+ZoxwrvNFX/mC4
PZEUw32Cw/vilJv1d0qAuk5oxXYMGVCzmhGSwbv/pap4ECjgjHaMhAoGHRvk0086
q2uJTq6hjsmpxSUhRkEvSqbdaTGLyGI5HZLjaWJjg7mbj6TAcbecJhOITzYzVIat
ILjfAV3o/kCNSDptLa9Zxf46jdjvZWc+22+l3R8Ci8S5y9WT/x38Hj2CbLSD9Lqe
jAQvamt5oucQ7rZ7L3ypQPURxrjNY/AhivYzroYlFYoAsp6J7Ni0FOMMUvh0mBVv
n49HUz7Hh/FEFlmYCA5vyBArSlB+VBzGdMbyc5xPjsKbwGHq24OgJRqWLFC84DiY
jnXERLU0GnopPhth/xhs
=wGXd
-END PGP SIGNATURE-

Re: [gentoo-portage-dev] [PATCH] dispatch-conf: fix popen UnicodeDecode error (bug 576788)

[Alexander Berntsen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

LGTM, but remember the plan about merging repoman first.

- -- 
Alexander
berna...@gentoo.org
https://secure.plaimi.net/~alexander
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCgAGBQJW5pJWAAoJENQqWdRUGk8Bk6EP/2LYCPnx+CsDAAe5I59MsVPf
yVj9zVN3AeNsPVkUThJ+FEbpZGXkEE3FYitD9EFRE3MJiVBSvDjZi4tWb39SsBXh
/N+Wy65Pg3uUEKODm8/9f1Vf72LTYWkY6xKciM1H+V6Z9UvWOt1D7+425siifeaJ
Ukkw9RgECg+cMLOejJqb6b6SNSnlK7miKpK8GwwFNUjtH11HLPjM8/XSI9fLJXU3
1QfYifwzssYKAQLCEum54HjgdQznfKyq6jB/34AFsmqLHbhyo05Efmox7efiSiTX
Cbf8gGlb2q7rstv8YlMWAV4dYWbcmBb+6HQeDb1h8S0WZBk5/3bYNKP9GPWA1g+i
nRSehATNtHc2QKQxOKPWQAPl9poVPgpJPA+MO4AVH+LpQCPjLuz8dZlXjRhW8w9p
Of1sh9gk25eQdKLCtUKPir+vaTC6E4m+lirG5aCN6xjH5CENB65AtHjNd8vCBgiB
Xcn6puWNQLCuhG3Tn9cYTmYSWiIj04tF6zr8wZn4fwSiV8Xs0xIMDwN6bgFSjy5z
c/noRm1xxTtKZ3w8TvuSmFvwkQxlT41gqPPneRd8xSK5Qtbm8MFFjVK+AQFhGOEx
lxWQv157k7f8D2wzhOH2fukkC1toubTwkaoZw1hULFOHR5inDR3m/rFRk9N3UMDM
BloEPxWwA1oSTIj05p0c
=enZC
-END PGP SIGNATURE-

Re: [gentoo-portage-dev] [Patch] Repoman rewrite stage2 modularization conversion complete

[Brian Dolbec]

On Mon, 14 Mar 2016 10:22:11 -0700
Zac Medico  wrote:

> On 03/14/2016 10:14 AM, Zac Medico wrote:
> > On 03/05/2016 01:37 PM, Brian Dolbec wrote:
> >   
> >> Zac, I'm done with code changes in the rewrite.  Ready for a last
> >> look before a merge.  Can you have a look again?  I did some
> >> changes/fixes and rebased them in.  Floppym hasn't reported any
> >> more bugs, so I think it's ready for broader testing in a
> >> release.  Then we can work on moving all the test data to a
> >> separate file in the tree or downloaded...  
> > 
> > The dynamic_data stuff in Scanner is a little hard to follow. Then
> > it calls dynamic_data.update(rdata), is there any chance that the
> > update operation might clobber something that shouldn't have been
> > clobbered?  
> 
> To clarify my question, suppose that one function returns {'foo':
> True} and another one returns {'foo', False}, so now there first
> {'foo': True} setting is forgotten. Is that going to be a problem?

No, as stated in my other reply.  There are only a few things that are
modified.  Mostly as I made a new module, following the original
order the checks were run.  As data was discovered missing it was added
to dynamic_data from the previous check that supplied it to the Scanner
class.  So, only data needed later was passed back to update the
dynamic_data.

Also all those checks originally ran in one huge 1k LOC loop with
another slightly smaller ebuild loop nested inside it.  So all those
variables were subject to change already by previous code run.  In the
stage1 rewrite, I/we did the same thing in creating the separated
checks classes.  After the check was done, only the data required was
brought back into the primary loop.

-- 
Brian Dolbec 

Re: [gentoo-portage-dev] [Patch] Repoman rewrite stage2 modularization conversion complete

[Zac Medico]

On 03/14/2016 10:14 AM, Zac Medico wrote:
> On 03/05/2016 01:37 PM, Brian Dolbec wrote:
> 
>> Zac, I'm done with code changes in the rewrite.  Ready for a last look
>> before a merge.  Can you have a look again?  I did some changes/fixes
>> and rebased them in.  Floppym hasn't reported any more bugs, so I think
>> it's ready for broader testing in a release.  Then we can work on
>> moving all the test data to a separate file in the tree or downloaded...
> 
> The dynamic_data stuff in Scanner is a little hard to follow. Then it
> calls dynamic_data.update(rdata), is there any chance that the update
> operation might clobber something that shouldn't have been clobbered?

To clarify my question, suppose that one function returns {'foo': True}
and another one returns {'foo', False}, so now there first {'foo': True}
setting is forgotten. Is that going to be a problem?
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] dblink: add locks for parallel-install with blockers (bug 576888)

[Brian Dolbec]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Mon, 14 Mar 2016 11:26:23 +0100
Alexander Berntsen  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
> 
> I can't say much more than "ACK, probably makes sense" really. But
> please test this *a lot* before merging it.
> 

I ack as well, the code looks good.  I don't know enough about to be
able to critique it in detail ;).   But it does look decent and the idea
of what it is doing sounds good.


> Regarding the merging of this patch, and th egencache patch that has
> already been released: I thought we agreed that .29 should be *only*
> the repoman merger, and then bug fixes go into a .30 where we try to
> get a stable release with the new repoman. Why was egencache merged
> anyway? Should we not merge repoman to stable ASAP before doing
> anything else? That would make .29 easier.
> - -- 
> Alexander
> berna...@gentoo.org
> https://secure.plaimi.net/~alexander

With a .29 release coming out very soon after the the .28, the .28
would not get much more testing for the stabilization.  If only the
repoman code was changed, it makes it easier to know that any bugs
submitted for .29 that re not repoman specific, apply to .28 as well.
 But more that if no non-repoman bugs were filed, then that clears .28
 for stabilization.

- -- 
Brian Dolbec 

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.1

iQJ8BAEBCgBmBQJW5wSjXxSAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNUQ3Qzc0RTA4MUNDNzBEQjRBNEFBRjVG
QkJEMDg3Mjc1ODIwRUQ4AAoJEPu9CHJ1gg7YF9IP/0s3s3h1eFWrr6bLfNOdDGvU
nfpnG97ulawslssJ4tL/oWFGmGaSqknUBl2rha65IhgeaswMGfmX1nHvyBqyf823
FgJMi+KZcB0kM3x63lEtB9K719tO9wF17JckmuIBW/4NLYAN/iHd7TKDHWyY1Isi
jStFNciRUY+h0IegnMqaNSfjPZAd0cjtxZlTO7tqbWQ9LH5ld90bZ84PgPp03poT
0/ZaYX4szi5e8FKhm7CThyZhYrYmcX0OpEDuSuLCUAwjWTYbhePXNOvlr0lC98ZG
itafmalGoKrpbsAsqN13BDuf0a053NKAePjveI4Gt5RYK7UiFs6G7sdCe14zq0SL
bePKF6aMc+PUyWaaF5vFywuDokjVfc6fyf/4EEpH90VRiIvTZlyMcIhClBgRsk57
sslUFm3lMLR1gW4QPxswn5ALtFt16+X7s/Zu/XIsZMYU0XJ1BM8N1/twHBx79O0G
RVRiRQPU1B+YhT9psCab4RZzhY9JqQZRdSV7mrBCBLAlKMF15PH7d2OSlGVweKSL
AjKa3ufGwy+KyCGqNQjYeYYFhW/F7NxfjASLg+/j4mCx1+OrCGzvXWuL/xMRszCP
5+KpEtdx+hQe4J0OufJypPYwVXpikpfoyW7oa44JyChuW/kzz1ESMr2e3vn07EJX
i3hjWpmUlCxgdWsWEKn8
=9IBu
-END PGP SIGNATURE-

Re: [gentoo-portage-dev] [Patch] Repoman rewrite stage2 modularization conversion complete

[Brian Dolbec]

On Mon, 14 Mar 2016 17:18:27 -0700
Zac Medico  wrote:

> On 03/14/2016 10:52 AM, Brian Dolbec wrote:
> > On Mon, 14 Mar 2016 10:22:11 -0700
> > Zac Medico  wrote:
> >   
> >> On 03/14/2016 10:14 AM, Zac Medico wrote:  
> >>> On 03/05/2016 01:37 PM, Brian Dolbec wrote:
> >>> 
>  Zac, I'm done with code changes in the rewrite.  Ready for a last
>  look before a merge.  Can you have a look again?  I did some
>  changes/fixes and rebased them in.  Floppym hasn't reported any
>  more bugs, so I think it's ready for broader testing in a
>  release.  Then we can work on moving all the test data to a
>  separate file in the tree or downloaded...
> >>>
> >>> The dynamic_data stuff in Scanner is a little hard to follow. Then
> >>> it calls dynamic_data.update(rdata), is there any chance that the
> >>> update operation might clobber something that shouldn't have been
> >>> clobbered?
> >>
> >> To clarify my question, suppose that one function returns {'foo':
> >> True} and another one returns {'foo', False}, so now there first
> >> {'foo': True} setting is forgotten. Is that going to be a
> >> problem?  
> > 
> > No, as stated in my other reply.  There are only a few things that
> > are modified.  Mostly as I made a new module, following the original
> > order the checks were run.  As data was discovered missing it was
> > added to dynamic_data from the previous check that supplied it to
> > the Scanner class.  So, only data needed later was passed back to
> > update the dynamic_data.
> > 
> > Also all those checks originally ran in one huge 1k LOC loop with
> > another slightly smaller ebuild loop nested inside it.  So all those
> > variables were subject to change already by previous code run.  In
> > the stage1 rewrite, I/we did the same thing in creating the
> > separated checks classes.  After the check was done, only the data
> > required was brought back into the primary loop.
> >   
> 
> I've found what may be a real instance of the kind of problem I was
> worried about. The 'allvalid' key is set in both
> pym/repoman/modules/scan/ebuild/isebuild.py and
> pym/repoman/modules/scan/ebuild/ebuild.py, and its non-trivial for me
> to determine whether this is a real problem or not.

It is not a problem.

allvalid is initialized in the isEbuild class at the start of the pkg
level checks to True, then there are several things that if found reset
it to False...

Then when it gets to the ebuild level checks the ebuild check there
just sets it False if the pkg.invalid variable is True.  In some cases
it might be a double False, but never will it trounce the previous
setting.

The only consumer for that allvalid variable is the metadata
UnusedCheck class.  

So the allvalid variable is True until found False
by whichever checks along the way find it to be False.  Like a fuse,
it's good until it's blown, then it can never be good again.  I don't
think this particular variable justifies a special class that more
fully mimics a fuse.  Impossible to reset it like a breaker.

To be honest I did not look into the pkg.invalid variable's need to be
setting it False in the Ebuild class.  It may in fact be a dupe of a
setting in the isEbuild class or it might be moved there. That original
spaghetti code could in fact have had duplicates since it was so hard
to figure out where to embed something.
-- 
Brian Dolbec 

Re: [gentoo-portage-dev] [Patch] Repoman rewrite stage2 modularization conversion complete

[Zac Medico]

On 03/14/2016 05:47 PM, Brian Dolbec wrote:
> On Mon, 14 Mar 2016 17:18:27 -0700
> Zac Medico  wrote:
> 
>> On 03/14/2016 10:52 AM, Brian Dolbec wrote:
>>> On Mon, 14 Mar 2016 10:22:11 -0700
>>> Zac Medico  wrote:
>>>   
 On 03/14/2016 10:14 AM, Zac Medico wrote:  
> On 03/05/2016 01:37 PM, Brian Dolbec wrote:
> 
>> Zac, I'm done with code changes in the rewrite.  Ready for a last
>> look before a merge.  Can you have a look again?  I did some
>> changes/fixes and rebased them in.  Floppym hasn't reported any
>> more bugs, so I think it's ready for broader testing in a
>> release.  Then we can work on moving all the test data to a
>> separate file in the tree or downloaded...
>
> The dynamic_data stuff in Scanner is a little hard to follow. Then
> it calls dynamic_data.update(rdata), is there any chance that the
> update operation might clobber something that shouldn't have been
> clobbered?

 To clarify my question, suppose that one function returns {'foo':
 True} and another one returns {'foo', False}, so now there first
 {'foo': True} setting is forgotten. Is that going to be a
 problem?  
>>>
>>> No, as stated in my other reply.  There are only a few things that
>>> are modified.  Mostly as I made a new module, following the original
>>> order the checks were run.  As data was discovered missing it was
>>> added to dynamic_data from the previous check that supplied it to
>>> the Scanner class.  So, only data needed later was passed back to
>>> update the dynamic_data.
>>>
>>> Also all those checks originally ran in one huge 1k LOC loop with
>>> another slightly smaller ebuild loop nested inside it.  So all those
>>> variables were subject to change already by previous code run.  In
>>> the stage1 rewrite, I/we did the same thing in creating the
>>> separated checks classes.  After the check was done, only the data
>>> required was brought back into the primary loop.
>>>   
>>
>> I've found what may be a real instance of the kind of problem I was
>> worried about. The 'allvalid' key is set in both
>> pym/repoman/modules/scan/ebuild/isebuild.py and
>> pym/repoman/modules/scan/ebuild/ebuild.py, and its non-trivial for me
>> to determine whether this is a real problem or not.
> 
> It is not a problem.
> 
> allvalid is initialized in the isEbuild class at the start of the pkg
> level checks to True, then there are several things that if found reset
> it to False...
> 
> Then when it gets to the ebuild level checks the ebuild check there
> just sets it False if the pkg.invalid variable is True.  In some cases
> it might be a double False, but never will it trounce the previous
> setting.

Makes sense, thanks for the explanation.

> The only consumer for that allvalid variable is the metadata
> UnusedCheck class.  
> 
> So the allvalid variable is True until found False
> by whichever checks along the way find it to be False.  Like a fuse,
> it's good until it's blown, then it can never be good again.  I don't
> think this particular variable justifies a special class that more
> fully mimics a fuse.  Impossible to reset it like a breaker.

Yeah, let's do it. It's a great opportunity to add clarity to the code,
and prevent future goofs.

> To be honest I did not look into the pkg.invalid variable's need to be
> setting it False in the Ebuild class.  It may in fact be a dupe of a
> setting in the isEbuild class or it might be moved there. That original
> spaghetti code could in fact have had duplicates since it was so hard
> to figure out where to embed something.
> 

It's not a dupe. The Package.invalid property is used to implement
various checks that are not duplicated elsewhere.
-- 
Thanks,
Zac