[Bug 59930] git fetches timing out, also cause bogus jenkins failures
https://bugzilla.wikimedia.org/show_bug.cgi?id=59930 Antoine "hashar" Musso changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |WORKSFORME --- Comment #7 from Antoine "hashar" Musso --- I haven't noticed such issue for a few weeks now. Assuming some workspace got corrupted and caused git to choke on it. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 59930] git fetches timing out, also cause bogus jenkins failures
https://bugzilla.wikimedia.org/show_bug.cgi?id=59930 --- Comment #6 from Antoine "hashar" Musso --- I looked at the failing jobs of all Gerrit changes mentioned above. All of them had failure of mediawiki-core-phpunit job when it was being run on lanthanum server and in workspace above. I guess it ended up badly initialized somehow. I am not sure what is the root cause though. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 59930] git fetches timing out, also cause bogus jenkins failures
https://bugzilla.wikimedia.org/show_bug.cgi?id=59930 --- Comment #5 from Antoine "hashar" Musso --- Delete /srv/ssd/jenkins-slave/workspace/mediawiki-core-phpunit-misc@3 workspace on lanthanum. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 59930] git fetches timing out, also cause bogus jenkins failures
https://bugzilla.wikimedia.org/show_bug.cgi?id=59930 Antoine "hashar" Musso changed: What|Removed |Added Component|Git/Gerrit |Continuous integration --- Comment #4 from Antoine "hashar" Musso --- (In reply to comment #2) > https://gerrit.wikimedia.org/r/103546 , again. > > Can we please revert jenkins to when it worked? That is unrelated to the upgrade of Zuul I did last week which is "simply" add a middle war between Zuul and Jenkins: Gearman. The timeout issue is not related to Gerrit since we do not use it. The changes are fetched using something like: git fetch \ refs/zuul/master/Z0e1e8799e33145bc911d2bd465d59179 \ git://zuul.eqiad.wmnet/mediawiki/core \ --reference=/srv/ssd/gerrit/mediawiki/core.git The URL git://zuul.eqiad.wmnet/mediawiki/core points to gallium server which has the Zuul daemon. That is where the merge references are created. They are publishing using git-daemon. The reference /srv/ssd/gerrit/mediawiki/core.git is a replication of git repositories which is on the same disk as the jobs workspace (/srv/ssd). Which mean when cloning, git will use hardlinks and save a ton of network I/O and disk space. When the fetch occurs, the client side apparently send to the server (git://zuul.eqiad.wmnet/) a list of all objects then a diff is made server side and the missing elements are sent back to the client. The timeouts might be caused when a new workspace is created which cause the full repository to be sent to the client. Might end up taking longer than 10 minutes :/ -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 59930] git fetches timing out, also cause bogus jenkins failures
https://bugzilla.wikimedia.org/show_bug.cgi?id=59930 Bartosz Dziewoński changed: What|Removed |Added CC||christ...@quelltextlich.at, ||marcin.cies...@gmail.com, ||rlan...@gmail.com Component|Continuous integration |Git/Gerrit Summary|Bogus jenkins failures |git fetches timing out, ||also cause bogus jenkins ||failures --- Comment #3 from Bartosz Dziewoński --- The root cause it probably the fact that gerrit itself is incredibly slow for the last few days and so the fetches keep timing out. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
