On 04/03/15 14:29, liuxinguo wrote:
The network of our CI is poor so sometime the zuul will get stuck when
fetching patch from gerrit. When running “ps -ef|grep zuul” I find it
get stuck like this:
root@3rd-ci-master:/var/log/zuul# ps -ef|grep zuul
root 698 9074 0 16:48 pts/0 00:00:00 grep --color=auto zuul
zuul 31730 1 2 03:01 ? 00:16:37 /usr/bin/python
/usr/local/bin/zuul-server
zuul 31739 1 0 03:01 ? 00:00:00 /usr/bin/python
/usr/local/bin/zuul-merger
zuul 31748 31730 0 03:01 ? 00:00:00 /usr/bin/python
/usr/local/bin/zuul-server
zuul 32008 31739 0 05:08 ? 00:00:00 git remote update origin
zuul 32009 32008 0 05:08 ? 00:00:00 git fetch --multiple
origin
zuul 32010 32009 0 05:08 ? 00:00:00 git fetch --append origin
zuul 32011 32010 0 05:08 ? 00:00:00 /bin/bash
/var/lib/zuul/git/.ssh_wrapper -p 29418
[email protected]
<mailto:[email protected]> git-upload-pack
'/openstack/cinder'
zuul 32012 32011 0 05:08 ? 00:00:00 ssh -i
/var/lib/zuul/ssh/id_rsa -p 29418
[email protected]
<mailto:[email protected]> git-upload-pack
'/openstack/cinder'
I have to add a Crontab to restart the zuul and zuul-merge service and
kill all these stucked progesses or do this manually.
·Is there any better method to avoid the “git remote update origin”
and “ssh_wrapper” getting stuck when the network is poor.
Thanks for any input!
Hello,
The zuul-merger ensures the repository is up-to-date before attempting
to merge the proposed patch. It indeeds uses 'git remote update'. git
sends a list of objects to the server which let the server send you only
the missing objects.
Since the zuul-merger repositories have a reference created for each
patch merged, they are all sent to Gerrit which takes quite a while. On
Wikimedia setup it took up to 20 seconds for one of our biggest repos
see https://phabricator.wikimedia.org/T70481 for details.
I proposed a tiny python script which let one clear out old references
from the zuul-merger repositories. It looks at zuul refs and delete them
when the commit date is older than a given amount of days.
https://review.openstack.org/#/c/109276/
Safe example usage:
zuul_clear_refs.py --verbose --dry-run --until 90 /srv/zuul/git/project
Note that "git remote update" is really "git fetch --all". It might be
possible to limit the ref between fetched, ie ignore the zuul ones when
processing.
You probably want to have a cronjob which repack all the ref in a single
pack ie:
find /var/lib/zuul/git/ -maxdepth 3 -type d -name ".git" -exec git
--git-dir="{}" pack-refs --all \;
Taken from Openstack puppet manifest for the zuul mergers:
https://github.com/openstack-infra/puppet-zuul/blob/0f585bef4f7822975/manifests/merger.pp#L27-L35
Antoine Musso
_______________________________________________
OpenStack-Infra mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
