hashar added a comment.
I have done various hacking here and there nothing conclusive. I have rolled back: - https://integration.wikimedia.org/ci/job/integration-quibble-fullrun/ - Dockerd --- Dockerd running with `docker: true` gives a lot more logs in the journal. At the end of the run, the container is reaped as far as dockerd knows about. The issue seems to be betwen the docker run client and the docker daemon. Some extra finding: we run the containers with `--init`, that instructs Docker to spawn the container with `docker-init` as PID #1 and it is in charge of passing signals and killing all process / zombies etc. Passing `TINI_VERBOSITY=3` to the container gives a bit more information. Example of me killing php: 13:37:57 INFO:backend.MySQL:Terminating MySQL # expected 2 seconds wait 13:37:59 subprocess.CalledProcessError: Command '['php', 'tests/phpunit/phpunit.php', '--group', 'Database', '--exclude-group', 'Broken,ParserFuzz,Stub,Standalone', '--log-junit', '/workspace/log/junit-db.xml']' returned non-zero exit status -15 13:37:59 [INFO tini (1)] Spawned child process '/src/utils/ci-fullrun.sh' with pid '7' 13:37:59 [DEBUG tini (1)] Received SIGCHLD .... 13:37:59 [DEBUG tini (1)] Reaped child with pid: '267' # long wait which is the subject of this bug 13:38:05 + forensics ... `forensics` is a result of a `trap forensics EXIT` in the shell script that invokes `docker run`. The above definitely confirms to me that it is an issue with docker run not exiting immediately after the container has finished. I have no idea how to further debug that though. Maybe the docker command line client has some logging/debugging. I looked for the sources but couldn't find them under https://github.com/moby/moby TASK DETAIL https://phabricator.wikimedia.org/T265615 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hashar Cc: Daimona, thcipriani, hashar, awight, kostajh, Lucas_Werkmeister_WMDE, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Liudvikas, Scott_WUaS, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
