hashar added subscribers: kostajh, awight, hashar.
hashar edited projects, added Continuous-Integration-Infrastructure,
Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)),
Release-Engineering-Team (CI & Testing services); removed
Continuous-Integration-Config.
hashar added a comment.
The relevant code:
name=quibble/backend.py
class BackendServer:
...
def stop(self):
if self.server is not None:
self.log.info('Terminating %s', self.__class__.__name__)
self.server.terminate()
try:
self.server.wait(2)
except subprocess.TimeoutExpired:
self.server.kill() # SIGKILL
finally:
self.server = None
Where `self.server` is a Popen instance
<https://docs.python.org/3/library/subprocess.html#subprocess.Popen>. So we
basically send `SIGTERM` and have a busy loop waiting for two seconds which
should then send `SIGKILL`. Clearly that does not work as expected :-\
The python documentation for `Popen.wait()` mentions it can deadlock when
sending stdout/stderr to pipes:
> Note
>
> This will deadlock when using stdout=PIPE or stderr=PIPE and the child
process generates enough output to a pipe such that it blocks waiting for the
OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to
avoid that.
But we use /dev/null when we start the server:
self.server = subprocess.Popen([
'/usr/sbin/mysqld', # fixme drop path
'--skip-networking',
'--datadir=%s' % self.rootdir,
'--log-error=%s' % self.errorlog,
'--pid-file=%s' % self.pidfile,
'--socket=%s' % self.socket,
],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
Some stuff we can try:
- reproduce it obviously
- check whether MariaDB version has changed in the CI image
- add some logging in the stop command (cause clearly
`subprocess.TimeoutExpired` should be raised and cause a SIGKILL to be send)
- add some option to capture MySQL stdout/stderr (using `stream_relay` which
can consumes a PIPE and output it via `logging`).
TASK DETAIL
https://phabricator.wikimedia.org/T265615
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: hashar, awight, kostajh, Lucas_Werkmeister_WMDE, Akuckartz, darthmon_wmde,
Nandana, NebulousIris, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer,
_jensen, rosalieper, Liudvikas, Scott_WUaS, thcipriani, Wikidata-bugs, aude,
Jdforrester-WMF, Mbch331, Jay8g, Totolinototo3, Redabr4, Zanziii, Sadisticturd,
A.S.Kochergin
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
