hashar added a comment.
When trying the jobs yesterday they failed again. The reason is that the
extension dependencies are not injected by Zuul since the change is in progress
and not deployed. I have been testing them by manually triggering them on
Jenkins, I have added a temp change
hashar closed this task as "Resolved".
hashar claimed this task.
hashar added a comment.
I have tested https://gerrit.wikimedia.org/r/c/integration/config/+/790621
against https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/767776
the jobs work with Quibble 1.4.5 +
hashar added a comment.
@Lucas_Werkmeister_WMDE maybe that can be reproduced on `testwikidatawiki`
(no clue about the domain / URL though).
TASK DETAIL
https://phabricator.wikimedia.org/T307586
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
hashar added a comment.
I am not sure whether anything is actionable for this task. It got filed in
August 2021 and a lot of things have changed since then (Quibble improvements,
change of the underlying WMCS and Docker infra). The Wikibase test suites are
definitely heavy though and we
hashar added a comment.
@kostajh has found the issue, the failing jobs are the one that do not start
supervisord and thus do not have memcached server running. Quibble 1.4.4
configure the caches to use memcached whenever the PHP extension is loaded but
since there is no memcached server
hashar added a comment.
I supposed the code is somewhere in
https://github.com/dependabot/dependabot-core/ but I could not find out how
`---\n` is injected :-\
TASK DETAIL
https://phabricator.wikimedia.org/T295601
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
hashar added a comment.
The hook in upstream is at
https://gerrit.googlesource.com/gerrit/+/refs/heads/master/resources/com/google/gerrit/server/tools/root/hooks/commit-msg
(git log
<https://gerrit.googlesource.com/gerrit/+log/refs/heads/master/resources/com/google/gerrit/server/tools/r
hashar added a comment.
I am not quite sure what happened with the npm cache so I have nuked the
cache for both jobs.
npm reported them as being corrupted, I thought about T295351
<https://phabricator.wikimedia.org/T295351> but that one is caused when some
files have zero bytes.
hashar closed subtask T299211: Request increased quota for integration Cloud
VPS project as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T265615
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: aborrero, dcaro, Aklapper, dduvall
hashar closed this task as "Resolved".
hashar claimed this task.
hashar added a comment.
Solved! Apparently the Quibble 1.3.0 images were the sole affected, I still
have rebuild the rest of the images for good measure :D
TASK DETAIL
https://phabricator.wikimedia.org/T299
hashar added a comment.
It is fixed by https://gerrit.wikimedia.org/r/c/integration/config/+/754905
which would stop getting php8.1 being installed. I have verified the new images
have the proper php version:
Successfully published image
docker-registry.discovery.wmnet/releng
hashar added a comment.
In T299389#7627900 <https://phabricator.wikimedia.org/T299389#7627900>,
@kostajh wrote:
> Can/should we use pecl to install apcu and xdebug? e.g. `pecl install apcu`
& `pecl install xdebug`
That is a bit of a pain, we would need various
hashar updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T299389
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: hashar, kostajh, Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657,
Invadibot, Lalamarie69
hashar added a comment.
From `apt list`:
| php-apcu| 5.1.21+4.0.11-7+0~20220117.31+debian10~1.gbp9a887a
|
| php-common | 2:92+0~20220117.43+debian10~1.gbpe0d14e
|
| php-xdebug |
3.1.2+2.9.8+2.8.1+2.5.5-1+0~20220117.45+debian10~1
hashar added subscribers: kostajh, hashar.
hashar added a project: Continuous-Integration-Infrastructure.
hashar triaged this task as "High" priority.
hashar added a comment.
I have upgraded the Jenkins jobs this morning to use Quibble 1.3.0 which has
patches by @kostajh . I have th
hashar added a subtask: T299211: Request increased quota for integration Cloud
VPS project.
TASK DETAIL
https://phabricator.wikimedia.org/T265615
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: aborrero, dcaro, Aklapper, dduvall, bd808
hashar added subscribers: dcaro, aborrero.
hashar added a comment.
I have talked to @aborrero and @dcaro about the limits. They are inclined to
further tune the limit like has been done last year by Andrew ( T266777#6589231
<https://phabricator.wikimedia.org/T266777#6589
hashar added a comment.
@awight potentially though `tmpfs` goes in memory and is expensive. A run
with core + Wikibase and all the npm dependencies turn out to be around 9G iirc
which is a lot to put in RAM.
I was looking at a stall build today and it `iotop` definitely showed up
hashar added a comment.
I have found the issue!
I have created the Selenium jobs after the wikibase-client-docker and
wikibase-repo-docker phpunit jobs which invoke the
`extensions/Wikibase/build/jenkins/mw-apply-wb-settings.sh` with `-r client` or
`-r repo` respectively. For Selenium
hashar added a comment.
I got the `repo` variant passing by adding a few extensions as dependencies:
| mediawiki/skins/MinervaNeue|
| mediawiki/extensions/Cite |
| mediawiki/extensions/MobileFrontend|
| mediawiki/extensions
hashar added a comment.
In T287582#7368381 <https://phabricator.wikimedia.org/T287582#7368381>,
@Addshore wrote:
> I have a suspicion that we would not want to split out `client/data-bridge`
tests.
> data-bridge integrated directly with MediaWiki and should be gated,
hashar added a comment.
In T194299#4840260 <https://phabricator.wikimedia.org/T194299#4840260>,
@Addshore wrote:
> From yesterday:
>
> Wikibase\UpsertSqlIdGenerator::upsertId 10.64.48.26 1205Lock
wait timeout exceeded; try restarting transaction (10.64.4
hashar added a comment.
Maybe related is T194299 <https://phabricator.wikimedia.org/T194299>, there
are mentions of `$wgWBRepoSettings['idGenerator'] = 'mysql-upsert'`, I don't
know what the default is or what is set on the CI job.
The comment T194299#4840260
hashar added a parent task: T226869: Run browser tests in parallel.
TASK DETAIL
https://phabricator.wikimedia.org/T298682
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: Aklapper, kostajh, hashar, Invadibot, maantietaja, Akuckartz
hashar created this task.
hashar added projects: Wikidata, ci-test-error.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: wdwb-tech.
TASK DESCRIPTION
The webdriver.io Selenium tests are run serially. As part of T226869
<ht
hashar added a comment.
I have checked in Docker and it does a `os.removeAll` which is waited for
completion. If a container has huge amount of files that have to be deleted,
that would file the memory cache for writes which is is known in Linux as the
Dirty memory. From `proc(5
hashar added a comment.
Pfff I am not focusing today. The `INFO:backend.MySQL:Terminating MySQL` is
the last message send by Quibble and it has indeed completed (at max roughly 2
seconds after the message got written). What is taking age and got thoroughly
investigated previously
hashar added a comment.
Quibble terminates MySQL via `quibble.backend.Backend.stop()` which does:
def stop(self):
if self.server is not None:
self.log.info('Terminating %s', self.__class__.__name__)
self.server.terminate()
try
hashar added a comment.
Thank you @Urbanecm for fixing up the CI jobs!
The original patch is included in 1.38.0-wmf.6 which hence does not a
backport.
TASK DETAIL
https://phabricator.wikimedia.org/T293885
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
hashar closed this task as "Resolved".
hashar added a comment.
There was just a few of those happening in production (once per hour or so).
I have deployed the change to production. Thank you for the quick fix!
TASK DETAIL
https://phabricator.wikimedia.org/T293860
EMAIL P
hashar added a comment.
Out of curiosity, how has it been possible to get the faulty patch to pass
through our testsuite? Is that due to some race condition? Since patch to
mediawiki/core are tested against Wikibase a patch causing Wikibase to fail
should not pass.
TASK DETAIL
https
hashar added a parent task: T287216: ParserOutput should implement
ContentMetadataCollector.
TASK DETAIL
https://phabricator.wikimedia.org/T293860
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Lucas_Werkmeister_WMDE, hashar
Cc
hashar added a comment.
That has been deprecated as part of T287216: ParserOutput should implement
ContentMetadataCollector <https://phabricator.wikimedia.org/T287216> specially
by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/727402 . A code search
has definitely bee
hashar created this task.
hashar added projects: Wikimedia-production-error,
MediaWiki-extensions-WikibaseClient, Wikidata-Campsite.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
Error
-
- mwversion: `1.38.0-wmf.5`
- reqId: `b189c229-467f-484e-9b29
hashar closed this task as "Resolved".
hashar claimed this task.
hashar added a comment.
It was still an issue last time I checked but I have been unable to reproduce
it. I agree with Lucas, given it is harmless, there is no point in spending
more time on this. Thank you!
T
hashar reopened this task as "Open".
hashar added a comment.
Sorry @Lucas_Werkmeister_WMDE , I have missed your comment. @kostajh raised
it again today so reopening.
21:26:14 hashar: does it seem normal for
`INFO:backend.MySQL:Terminating MySQL` to take ~2 minutes? looking
hashar updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T291215
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: Michael, Aklapper, hashar, Invadibot, maantietaja, Akuckartz,
darthmon_wmde, Nandana, Lahi, Gq86
hashar created this task.
hashar added projects: Wikimedia-production-error, Wikidata-Campsite, Wikidata.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: wdwb-tech.
TASK DESCRIPTION
Error
-
- mwversion: `1.37.0-wmf.21`
- reqId
hashar moved this task from INBOX to Seen on the Release-Engineering-Team board.
hashar edited projects, added Release-Engineering-Team (Seen); removed
Release-Engineering-Team.
TASK DETAIL
https://phabricator.wikimedia.org/T281122
WORKBOARD
https://phabricator.wikimedia.org/project/board
hashar moved this task from INBOX to Next on the Release-Engineering-Team board.
hashar edited projects, added Release-Engineering-Team (Next); removed
Release-Engineering-Team.
TASK DETAIL
https://phabricator.wikimedia.org/T287582
WORKBOARD
https://phabricator.wikimedia.org/project/board
hashar triaged this task as "Low" priority.
hashar added a comment.
Thanks @Lucas_Werkmeister_WMDE as far as I can tell that is the proper git
and v2 is usually nice. The update script looks really straightforward:
cd ${GIT_PATH}
git pull origin master 2>> &q
hashar added a project: Structured-Data-Backlog.
hashar added a comment.
That is in WikibaseMediaInfo and I had added Wikidata/WMDE to the task due to
the Wikibase prefix. It seems the extension is instead maintained by the
Structured Data team.
The key collision is due to space versus
hashar closed this task as "Declined".
hashar added a comment.
If Wikibase has a breaking change, the other extensions should have patches
written to adjust and the Wikibase change can then be marked as depending on
the others. This way the integration tests pass. Then on
hashar added a comment.
I have renamed the bridge Jenkins job to client. Amended the configuration
to run `extensions/Wikibase/build/jenkins/mw-apply-wb-settings.sh` with either
`-r client` or `-r repo`.
`selenium-test:repo`
https
hashar added a subscriber: RKemper.
hashar added a comment.
The timeout for
https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-docker/
has been raised from 45 minutes to 60 minutes which should give it enough time
to complete. The saved maven cache would
hashar created this task.
hashar added projects: Wikidata, Wikidata-Campsite, Gerrit.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
While browsing Gerrit logs on logstash
<https://logstash.wikimedia.org/app/dashboards#/view/AW1f-0k0ZKA7RpirlnKV>, I
found out tw
hashar claimed this task.
hashar edited projects, added Release-Engineering-Team (Doing); removed
Release-Engineering-Team.
hashar added a comment.
The two root causes are:
- archiva.wikimedia.org has very poor latency/bandwith. I have filed a task
about it but haven't raised much
hashar closed subtask T220763: Rebuild CI Docker images to drop ENV
LANG=en_US.UTF-8 LANGUAGE=en_US:en
LC_ALL=en_US.UTF-8 as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T210260
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: akosiaris
hashar added a comment.
An example is `mediawiki/libs/metrics-platform.git` which co host code for
NodeJs, PHP, Swift and Java, each in their own directory having the relevant
package manager file:
| JS |
https://gerrit.wikimedia.org/r/c/mediawiki/libs/metrics-platform/+/690049
hashar closed this task as "Declined".
TASK DETAIL
https://phabricator.wikimedia.org/T51236
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: hoo, aude, Ricordisamoa, Aklapper, Wikidata-bugs, hashar, Denny,
Lydia_Pintscher,
hashar closed subtask T51240: populateSitesTables.php should be run
automatically on new wiki addition as Declined.
TASK DETAIL
https://phabricator.wikimedia.org/T51236
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: hoo, aude
hashar closed this task as "Declined".
TASK DETAIL
https://phabricator.wikimedia.org/T51240
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: hoo, aude, Ricordisamoa, Aklapper, Wikidata-bugs, Krenair, hashar, Denny,
Lydia
hashar closed subtask T51240: populateSitesTables.php should be run
automatically on new wiki addition as Declined.
TASK DETAIL
https://phabricator.wikimedia.org/T51242
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: Wikidata-bugs, Denny
hashar closed subtask T51238: addWiki.php should have some hooks as
Declined.
TASK DETAIL
https://phabricator.wikimedia.org/T46749
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: Addshore, Denny, JeroenDeDauw, aude, Lydia_Pintscher
hashar closed subtask T51238: addWiki.php should have some hooks as
Declined.
TASK DETAIL
https://phabricator.wikimedia.org/T51240
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: hoo, aude, Ricordisamoa, Aklapper, Wikidata-bugs, Krenair
hashar closed this task as "Declined".
hashar added a comment.
Indeed that is the same, still tools / paws. From:
https://openstack-browser.toolforge.org/puppetclass/role::labs::lvm::mnt
> Project: tools
>
> tools-paws-*
>
> Project: toolsbeta
hashar closed this task as "Resolved".
hashar claimed this task.
hashar added a comment.
Closing as probably obsolete. It might have been related to the underlying
MySQL database being too slow at the time for whatever reasons. Nowadays it is
running with Innodb datafiles
hashar added a comment.
Danke Schon @Jakob_WMDE !
TASK DETAIL
https://phabricator.wikimedia.org/T279068
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ladsgroup, hashar
Cc: Jakob_WMDE, Michael, Addshore, Lucas_Werkmeister_WMDE, Aklapper
hashar added a comment.
@Ladsgroup thank you so much for fixing this test!
TASK DETAIL
https://phabricator.wikimedia.org/T279068
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ladsgroup, hashar
Cc: Michael, Addshore, Lucas_Werkmeister_WMDE
hashar renamed this task from "Wikibase data-client bridge selenium " to
"Wikibase data-client bridge selenium failure".
TASK DETAIL
https://phabricator.wikimedia.org/T279068
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To:
hashar added a comment.
Will try to figure out the reproduction command with the image used by the CI
job `docker-registry.wikimedia.org/releng/quibble-buster-php72:0.0.46-s1`
TASK DETAIL
https://phabricator.wikimedia.org/T279068
EMAIL PREFERENCES
https://phabricator.wikimedia.org
hashar created this task.
hashar added projects: Wikibase, Wikidata-Campsite.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
We are trying to update the CI jobs for Quibble from Stretch to Buster (which
bring in a new
hashar added a project: Wikibase.
Restricted Application added a project: wdwb-tech-focus.
TASK DETAIL
https://phabricator.wikimedia.org/T274506
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: Aklapper, hashar, Akuckartz, darthmon_wmde
hashar created this task.
hashar added projects: Wikimedia-production-error, Wikidata-Campsite.
hashar triaged this task as "High" priority.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
name=Error message
P
hashar added a comment.
I had dinner, come back and magic the task is closed. Thank you!
TASK DETAIL
https://phabricator.wikimedia.org/T273622
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Lucas_Werkmeister_WMDE, hashar
Cc: hashar, WMDE-leszek
hashar added a comment.
I guess we "just" need to backport those two merged changes to 1.36.0-wmf.29
? If so please cherry pick to `wmf/1.36.0-wmf.29` and either sync them or I
can do it.
1.36.0-wmf.29 is still blocked on other changes though so I am most probably
not goi
hashar added a comment.
Restricted Application added a project: wdwb-tech-focus.
Another one:
name=Unresolved redirect from Q104821940 to Q56305763
/w/rest.php/en.wikipedia.org/v3/page/pagebundle/Template%3AISO_15924%2Fqid/1003335501
Wikibase\DataModel\Services\Lookup
hashar added a comment.
Nice Lucas.
EventBus should be added to the wmf-quibble* jobs as well (T272863
<https://phabricator.wikimedia.org/T272863>) and PHPUnit tests currently fail
because of that string/array mismatch.
TASK DETAIL
https://phabricator.wikimedia.org/T272998
hashar added subscribers: Cparle, dcausse, matthiasmullie, hashar.
hashar added a comment.
The code comes from
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseMediaInfo/+/640820
(and depends on
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/640809
hashar added a comment.
Spotted again on commonswiki:
/srv/mediawiki/php-1.36.0-wmf.25/includes/libs/rdbms/database/Database.php:1680
counterexample
#0
/srv/mediawiki/php-1.36.0-wmf.25/includes/libs/rdbms/database/Database.php(1658):
Wikimedia\Rdbms\Database
hashar added a comment.
@Lucas_Werkmeister_WMDE congratulations on finding MW_API as being the root
cause!
TASK DETAIL
https://phabricator.wikimedia.org/T269608
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Lucas_Werkmeister_WMDE, hashar
Cc
hashar edited projects, added Continuous-Integration-Infrastructure; removed
Continuous-Integration-Infrastructure (Slipway).
TASK DETAIL
https://phabricator.wikimedia.org/T210286
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: Jakob_WMDE
hashar added a comment.
I don't have any lead myself beside the CentralAuth one (which does not seem
to change anything since my proposal fails still). It is late so yeah lets
check what we can do tomorrow and maybe loop other people. Thank you Luca!
TASK DETAIL
https
hashar added a comment.
Sorry Lucas I have misread your earlier comment about the failure occurring
with just Wikibase / FlaggedRevs. The test I have done reverting a CentralAuth
patch doesn't help indeed, that still fails :-\
TASK DETAIL
https://phabricator.wikimedia.org/T269608
EMAIL
hashar added a comment.
Next trick, download the Jenkins console log, filter for lines matching
`Prepared` to match the informational messages emitted by `zuul.cloner`. That
is the bit in Quibble which takes care of cloning the repositories, fetch the
patches / branch. Doing a diff yields
hashar added a comment.
It is very unlikely the Jenkins upgrade is any related. The jobs are running
in Docker containers and Jenkins is really just running `docker run` commands
over ssh to WMCS instances. I am not entirely ruling it out though, but it has
a very low chance
hashar added a comment.
Thank you Lucas and Amir for the investigation.
TASK DETAIL
https://phabricator.wikimedia.org/T269315
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: hashar, Lucas_Werkmeister_WMDE, Ladsgroup, mmodell, RhinosF1
hashar closed this task as "Resolved".
hashar added a comment.
That has been more or less fixed by raising the newish IO quota for the
integration instances. It is still not ideal and enhancements will be made
later on via the infrastructure task T266777
<https://phabricator.
hashar closed subtask T267388: Puppet failure on
deployment-memc08.deployment-prep.eqiad.wmflabs as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T213089
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: Ladsgroup, Krenair
hashar added a subtask: T267388: Puppet failure on
deployment-memc08.deployment-prep.eqiad.wmflabs .
TASK DETAIL
https://phabricator.wikimedia.org/T213089
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: Ladsgroup, Krenair, elukey, jijiki
hashar merged a task: T267075: Pipeline jobs freezing during teardown.
TASK DETAIL
https://phabricator.wikimedia.org/T265615
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: Aklapper, dduvall, bd808, jeena, Daimona, thcipriani, hashar
hashar merged a task: T267075: Pipeline jobs freezing during teardown.
hashar added subscribers: jeena, bd808, dduvall, Aklapper.
TASK DETAIL
https://phabricator.wikimedia.org/T265615
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc
hashar added a comment.
@Lucas_Werkmeister_WMDE yes Wikibase kind of highlights the issue since it
has so many files. I have also noticed the disk slowness on various other jobs
though, some started to timeout as a result due to the number of IO operations
they perform.
I thought
hashar added a subtask: T266777: integration instances suffer from high IO
latency due to Ceph.
TASK DETAIL
https://phabricator.wikimedia.org/T265615
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: Daimona, thcipriani, hashar, awight
hashar added a comment.
I have found out the root cause which is in the underlying infrastructure.
Will file a task about it tomorrow.
TASK DETAIL
https://phabricator.wikimedia.org/T265615
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
hashar added a comment.
I have looked at it live by running strace profiling against `dockerd` while
a container was being reaped:
name=strace -C -e trace=file -f -p `pidof dockerd`
% time seconds usecs/call callserrors syscall
hashar added a comment.
So my best assumption is that the source code / package install etc are done
inside the Docker container. Upon removal, Docker spends a lot of time to
delete all those files which might explain the issue. If that is true, that
can be reproduced by spawning
hashar added a comment.
Looks like the container is stuck in `removing` status. For a build with:
13:09:39 INFO:backend.MySQL:Terminating MySQL
13:10:33 [wmf-quibble-selenium-php72-docker] $ /bin/bash -xe
/tmp/jenkins4716707043969520657.sh
Running `docker container inspect
hashar added a comment.
No with child tracing (`-f`) and time spent in system call (`-T`). PID 23094
invokes `epoll_pwait` which resumes after 3 seconds for whatever reason.
name=strace -s 128 -tt -yy -T -f
[pid 23092] 10:21:51.923319 write(2,
"time=\"2020-10-28T10:21:5
hashar added a comment.
With `TINY_VERBOSITY=3` and `docker -D -l debug run`:
08:58:27 [INFO tini (1)] Main child exited normally (with status '1')
08:58:28 time="2020-10-28T08:58:28Z" level=debug msg="[hijack] End of
stdout"
08:58:38 Build step 'Execute
hashar added a comment.
I have done various hacking here and there nothing conclusive.
I have rolled back:
- https://integration.wikimedia.org/ci/job/integration-quibble-fullrun/
- Dockerd
---
Dockerd running with `docker: true` gives a lot more logs in the journal
hashar added a comment.
Same for Docker 19.03.13 / 1.3.7 . wmf-quibble-selenium-php72-docker builds
still have a minute delay or so.
TASK DETAIL
https://phabricator.wikimedia.org/T265615
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc
hashar added a comment.
I have upgraded containerd from 1.2.10-3 to 1.2.13-2 on eight hosts but that
unfortunately does not seem to help :-(
TASK DETAIL
https://phabricator.wikimedia.org/T265615
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
hashar added a comment.
For later reference:
11:23:24 INFO:backend.MySQL:Terminating MySQL
11:24:41 [wmf-quibble-selenium-php72-docker] $ /bin/bash -xe
/tmp/jenkins937915457220888713.sh
And in containerd the log shows the process got reaped properly?:
Oct 26 10:56:21
hashar added a comment.
I was watching integration-agent-docker-1004 which had 3 builds involving
Quibble. I have opened pages using the Jenkins monitoring interface that lets
one list processes and java threads. As soon as one showed the delay I
refreshed the two pages.
On the list
hashar claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T265615
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hashar
Cc: thcipriani, hashar, awight, kostajh, Lucas_Werkmeister_WMDE, Alter-paule,
Beast1978, Un1tY, Akuckartz, Hook696
hashar added a comment.
Next fun, digging into Jenkins!
I looked at
https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/64820/
which got affected by the same issue (all times are UTC):
counterexample
18:08:34 [wmf-quibble-selenium-php72-docker] $ /bin
hashar added a comment.
After many attempts, I finally got a lead
https://integration.wikimedia.org/ci/job/integration-quibble-fullrun/153/console
The job executes a shell script utils/ci-fullrun.sh in
integration/quibble.git, I have edited it to `echo CI FULLRUN completed` when
hashar added a comment.
I tried various combination and I can not reproduce it. Maybe it is race
condition between us sending SIGKILL (`self.server.kill()`) and python reaping
the processed on garbage collection when we set `self.server = None` :-\
TASK DETAIL
https
hashar added a comment.
Gave it a try with a bit more logging. One run with a 2 seconds delay which
is too short and trigger a SIGKILL:
00:22:05.191 INFO:quibble.commands:Shutting down backends:
00:22:05.191 INFO:backend.MySQL:Stopping MySQL
00:22:05.192
1 - 100 of 863 matches
Mail list logo
