BuildStream-Migration-Bot opened a new issue #1414: URL: https://github.com/apache/buildstream/issues/1414
[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/1414) In GitLab by [[Gitlab user @tristanvb]](https://gitlab.com/tristanvb) on Dec 10, 2020, 13:51 In [this recent pipeline](https://gitlab.com/BuildStream/buildstream/-/jobs/902850792), we can observe a spurious error from `tests/integration/cachedfail.py::test_push_failed_missing_shell[continue]` ``` Traceback (most recent call last): File "/builds/BuildStream/buildstream/src/buildstream/testing/runcli.py", line 386, in _invoke cli_object.main(args=args or (), prog_name=cli_object.name) File "/builds/BuildStream/buildstream/src/buildstream/_frontend/cli.py", line 238, in override_main original_main(self, args=args, prog_name=prog_name, complete_var=None, standalone_mode=standalone_mode, **extra) File "/builds/BuildStream/buildstream/.tox/py36/lib/python3.6/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/builds/BuildStream/buildstream/.tox/py36/lib/python3.6/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/builds/BuildStream/buildstream/.tox/py36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/builds/BuildStream/buildstream/.tox/py36/lib/python3.6/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/builds/BuildStream/buildstream/.tox/py36/lib/python3.6/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context().obj, *args, **kwargs) File "/builds/BuildStream/buildstream/src/buildstream/_frontend/cli.py", line 552, in show dependencies = app.stream.load_selection(elements, selection=deps, except_targets=except_) File "/builds/BuildStream/buildstream/src/buildstream/_stream.py", line 161, in load_selection load_artifacts=load_artifacts, File "/builds/BuildStream/buildstream/src/buildstream/_stream.py", line 1321, in _load self._pipeline.resolve_elements(self.targets) File "/builds/BuildStream/buildstream/src/buildstream/_pipeline.py", line 102, in resolve_elements element._initialize_state() File "/builds/BuildStream/buildstream/src/buildstream/element.py", line 1364, in _initialize_state self.__update_cache_keys() File "/builds/BuildStream/buildstream/src/buildstream/element.py", line 3250, in __update_cache_keys self.__update_artifact_state() File "/builds/BuildStream/buildstream/src/buildstream/element.py", line 3294, in __update_artifact_state self.__schedule_assembly_when_necessary() File "/builds/BuildStream/buildstream/src/buildstream/element.py", line 1619, in __schedule_assembly_when_necessary if not self.__should_schedule(): File "/builds/BuildStream/buildstream/src/buildstream/element.py", line 1588, in __should_schedule processing = self.__assemble_scheduled or self.__assemble_done or self._pull_pending() File "/builds/BuildStream/buildstream/src/buildstream/element.py", line 1871, in _pull_pending if self._cached() and self.__artifact._cache_key == self.__strict_cache_key: File "/builds/BuildStream/buildstream/src/buildstream/element.py", line 1185, in _cached return self.__artifact.cached() File "/builds/BuildStream/buildstream/src/buildstream/_artifact.py", line 582, in cached if not self._cas.contains_files(digests): File "/builds/BuildStream/buildstream/src/buildstream/_cas/cascache.py", line 155, in contains_files return len(self.missing_blobs(digests)) == 0 File "/builds/BuildStream/buildstream/src/buildstream/_cas/cascache.py", line 454, in missing_blobs response = cas.FindMissingBlobs(request) AttributeError: 'NoneType' object has no attribute 'FindMissingBlobs' ``` Suspicion is that this is a new race introduced by the threaded scheduler... some brief IRC log: ``` <tristan> juergbi, I think this is new: https://gitlab.com/BuildStream/buildstream/-/jobs/902850792 <tristan> It appears to be spurious failure where the internal CasCache's internal cas object might not be resolved on time <juergbi> hm, missing_blobs() calls get_cas(), which is supposed to do the right thing <juergbi> and this is in the main thread, afaict, so should also not be a thread-related race condition <tristan> https://gitlab.com/BuildStream/buildstream/-/snippets/2048752 <tristan> I thought it might be on my side (even I didnt play with that part)... <tristan> but running that test locally passes <tristan> tests/integration/cachedfail.py::test_push_failed_missing_shell[continue] <juergbi> looking at casdprocessmanager.py, I see a potential race condition <juergbi> in that get_cas() checks whether self._casd_channel is None but not whether self._casd_cas is None <tristan> The other tests in the same pipeline passed, save for fedora 31 which failed on a well known spurious error <tristan> So, this might not be because of your recent CASCache changes of course :) <tristan> Just thought, since you just just landed that, might be related <tristan> could be just very rare <juergbi> my changes shouldn't change anything with regards to establishing the connection, so I'd rather suspect the thread-based scheduler <juergbi> however, as this is not happening in a job thread, I also don't see right now how this can happen <juergbi> although, it might be that establish_connection() is running in a background thread (e.g. the usage monitor thread) <juergbi> we should definitely replace the casd_channel check with something that is not prone to this race condition <juergbi> can't be sure whether that's the issue here, though ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
