On Sun, 2022-04-17 at 14:41 +0100, Richard Purdie via lists.openembedded.org
wrote:
> On Sun, 2022-04-17 at 14:39 +0100, Richard Purdie via lists.openembedded.org
> wrote:
> > On Sat, 2022-04-16 at 23:28 +0100, Jose Quaresma wrote:
> > > With this LockedSet python class we can call the 'add' and 'remove'
> > > safely inside the ThreadedPool.
> > >
> > > This piece of code is taen from the stackoverflow
> > > https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe
> > >
> > > Fixes [YOCTO #14775] --
> > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775
> > >
> > > Signed-off-by: Jose Quaresma <[email protected]>
> > > ---
> > > meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++--
> > > 1 file changed, 22 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> > > index 1c0cae4893..a3ba748a1e 100644
> > > --- a/meta/classes/sstate.bbclass
> > > +++ b/meta/classes/sstate.bbclass
> > > @@ -918,8 +918,28 @@ sstate_unpack_package () {
> > > BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
> > >
> > > def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0,
> > > summary=True, **kwargs):
> > > - found = set()
> > > - missed = set()
> > > + #
> > > https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe
> > > + import threading
> > > + class LockedSet(set):
> > > + """A set where add(), remove(), and 'in' operator are
> > > thread-safe"""
> > > + def __init__(self, *args, **kwargs):
> > > + self._lock = threading.Lock()
> > > + super(LockedSet, self).__init__(*args, **kwargs)
> > > +
> > > + def add(self, elem):
> > > + with self._lock:
> > > + super(LockedSet, self).add(elem)
> > > +
> > > + def remove(self, elem):
> > > + with self._lock:
> > > + super(LockedSet, self).remove(elem)
> > > +
> > > + def __contains__(self, elem):
> > > + with self._lock:
> > > + super(LockedSet, self).__contains__(elem)
> > > +
> > > + found = LockedSet()
> > > + missed = LockedSet()
> > >
> > > def gethash(task):
> > > return sq_data['unihash'][task]
> >
> >
> > The series blew up on the autobuilder:
> >
> > testsdkest:
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/5079/steps/17/logs/stdio
> > oe-selftest:
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3461/steps/15/logs/stdio
> > and many more, see:
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3535
>
> That last one should be:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3539
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 7a18d9d554..99d125b8c5 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -1058,7 +1058,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False,
currentcount=0, summary=True,
connection_cache_pool = Queue(nproc)
checkstatus_init()
with concurrent.futures.ThreadPoolExecutor(max_workers=nproc)
as executor:
- executor.map(checkstatus, tasks)
+ executor.map(checkstatus, tasks.copy())
checkstatus_end()
bb.event.disable_threadlock()
fixes things.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#164579):
https://lists.openembedded.org/g/openembedded-core/message/164579
Mute This Topic: https://lists.openembedded.org/mt/90514091/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
