I found that paramiko does support proxies [1]. I remember looking at this a long time ago and you should be able to update zuul code to use it [2].
Ramy [1] http://blog.adimian.com/2014/10/paramiko-and-corporate-proxies/ [2] http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/lib/gerrit.py#n61 From: Tang Chen [mailto:[email protected]] Sent: Sunday, July 19, 2015 10:48 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [CI] How to set a proxy for zuul. Hi Asselin, Abhishek, Thanks for the reply. :) On 07/19/2015 02:41 AM, Asselin, Ramy wrote: HI Abhi & Tang, Sorry I missed this thread. Let me know if you’ve resolved your issues. My repo is undergoing migrations to reuse components in openstack-infra/puppet-openstackci. For single-use-nodes, the file you need has been removed here [1]: But I see now that it is still needed, or a different function is needed based on this version used by infra: [2]. I will explore a solution. A couple other notes, please use ci-sandbox [3] instead of sandbox. OK. Zuul use behind a proxy: seems you got past this? Could you share your solution? The root cause is that zuul uses a library named paramiko to create connection with low level socket APIs. And it doesn't provide proxy functionary. I tried to use tools like proxychain, to redirect the zuul connection to go through my proxy. But it doesn't work. If I use proxychain to run zuul service, it doesn't output anything to the log file, and the service will die soon. I think, there are two solutions: 1. Add proxy functionality to paramiko. 2. Add proxy functionality to zuul. That means, maybe, zuul does not use paramiko to create connection, but by itself. Solution 1 is much simpler, so for now, I just modified the source code of paramiko. I'm using python-socksipy package, and modified /usr/local/lib/python2.7/dist-packages/paramiko/client.py like this: diff --git a/client.py b/client.py index 15ff696..d7225ed 100644 --- a/client.py +++ b/client.py @@ -24,6 +24,7 @@ from binascii import hexlify import getpass import os import socket +import socks import warnings import pdb @@ -235,6 +236,7 @@ class SSHClient (ClosingContextManager): ``gss_deleg_creds`` and ``gss_host`` arguments. """ + """ if not sock: for (family, socktype, proto, canonname, sockaddr) in socket.getaddrinfo(hostname, port, socket.AF_UNSPEC, socket.SOCK_STREAM): if socktype == socket.SOCK_STREAM: @@ -251,6 +253,13 @@ class SSHClient (ClosingContextManager): except: pass retry_on_signal(lambda: sock.connect(addr)) + """ + + if not sock: + sock = socks.socksocket() + sock.setproxy(socks.PROXY_TYPE_SOCKS5, "MY_PROXY_IP", MY_PROXY_PORT, username="XXX", password="XXX") # This is review.openstack.org + addr = ('104.130.159.134', 29418) + retry_on_signal(lambda: sock.connect(addr)) t = self._transport = Transport(sock, gss_kex=gss_kex, gss_deleg_creds=gss_deleg_creds) t.use_compression(compress=compress) Of course, this is just a draft. It is only for my case, not for all. BTW, I'm now working on Fujitsu CI System, and really want to join into the development of openstack-infra. I think the proxy functionality is necessary for many companies. So if you are planing to add the proxy support, I think I can help. Thanks. :) Also, feel free to join 3rd party ci IRC meetings on freenode [4]. It’s a great place to ask questions and meet others setting up or maintaining these systems. Thanks, Ramy IRC: asselin [1] https://github.com/rasselin/os-ext-testing/commit/dafe822be7813522a6c7361993169da20b37ffb7 [2] https://github.com/openstack-infra/project-config/blob/master/zuul/openstack_functions.py [3] http://git.openstack.org/cgit/openstack-dev/ci-sandbox/ [4] http://eavesdrop.openstack.org/#Third_Party_Meeting From: Abhishek Shrivastava [mailto:[email protected]] Sent: Monday, July 13, 2015 11:51 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [CI] How to set a proxy for zuul. Also if you want to change it you will need to talk with Asselin Ramy who is the owner of the repo you followed. On Tue, Jul 14, 2015 at 12:18 PM, Abhishek Shrivastava <[email protected]<mailto:[email protected]>> wrote: Basically it is not required, and if you see /etc/jenkins_jobs/config folder you will find one dsvm-cinder-tempest.yaml which is to be used basically not examples.yaml. So its not an issue. On Tue, Jul 14, 2015 at 12:07 PM, Tang Chen <[email protected]<mailto:[email protected]>> wrote: On 07/14/2015 01:46 PM, Abhishek Shrivastava wrote: Instead of it use reusable_node option. Thanks. Problem resolved. :) BTW, single_use_node is written in layout.yaml by default. If it doesn't exist anymore, do we need a patch to fix it ? For someone who uses CI for the first time, it is really a problem...... And also, if I want to post patch for zuul, where should I post the patch ? Thanks. On Tue, Jul 14, 2015 at 9:12 AM, Tang Chen <[email protected]<mailto:[email protected]>> wrote: Hi Abhishek, All, I found the problem. My /etc/zuul/layout/layout.yaml has the following config: jobs: - name: ^dsvm-tempest.*$ parameter-function: single_use_node But in _parseConfig() in zuul/scheduler.py, it failed to find single_use_node(). fname = config_job.get('parameter-function', None) if fname: func = config_env.get(fname, None) if not func: raise Exception("Unable to find function %s" % fname) So projects section was not parsed. Does anyone know why ? Thanks. On 07/14/2015 10:54 AM, Tang Chen wrote: Hi Abhishek, I printed the self.layout.projects in zuul/scheduler.py, it is empty. So the project was not found. But I did do the jenkins-jobs --flush-cache update /etc/jenkins_jobs/config/ And I did configure openstack-dev/sandbox in layout.yaml. Do you have any idea what's wrong here ? Thanks. On 07/13/2015 05:58 PM, Tang Chen wrote: On 07/13/2015 04:35 PM, Abhishek Shrivastava wrote: Updating jobs using "sudo jenkins-jobs --flush-cache update /etc/jenkins_jobs/config/". Also update the myvendor in examples.yaml Sorry, I updated the jobs, restart the whole machine. But it still doesn't work. By the way, there is no vendor in examples.yaml. It is still this error: Project openstack-dev/sandbox not found Anything else should I pay attention to? Thanks. On Mon, Jul 13, 2015 at 1:45 PM, Tang Chen <[email protected]<mailto:[email protected]>> wrote: On 07/13/2015 03:50 PM, Abhishek Shrivastava wrote: Use tester or something, also are you updating the jobs or not? I used tester as my vendor. It doesn't work. And what do you mean by updating the jobs ? I built the noop-check-cimmunitication job once in Jenkins UI. Does it matter ? All the others are not touched. And referring to the error, "Project openstack-dev/sandbox not found", it seems like somewhere the project name was wrong. right ? Thanks. On Mon, Jul 13, 2015 at 1:16 PM, Tang Chen <[email protected]<mailto:[email protected]>> wrote: Hi Abhishek, Thanks for the quick reply. On 07/13/2015 03:16 PM, Abhishek Shrivastava wrote: Also check that Gearman is connecting or not through Jenkins UI. On Mon, Jul 13, 2015 at 12:45 PM, Abhishek Shrivastava <[email protected]<mailto:[email protected]>> wrote: First of all, change the "vendor" to your vendor name in /etc/jenkins_jobs/config/projects.yaml file. Also, restart the zuul and zuul merger. I did the check. Gearman plugin works find. In Jenkins UI, I tested the connection, and it succeeded. And also, I restarted zuul and zuul merger every time I modified the yaml files. But it doesn't work. And the vendor, does that matter ? And what vendor name should I provide ? I cannot find any vendor info in my Gerrit service account profile. For example, is XXX OK ? Thanks. On Mon, Jul 13, 2015 at 12:29 PM, Tang Chen <[email protected]<mailto:[email protected]>> wrote: Hi all, I have constructed my CI system. When I tested it with sandbox project, I posted a patch to Gerrit. https://review.openstack.org/201002 But I got this error in zuul scheduler: 2015-07-14 14:07:24,921 DEBUG zuul.Scheduler: Run handler awake 2015-07-14 14:07:24,921 DEBUG zuul.Scheduler: Fetching trigger event 2015-07-14 14:07:24,922 DEBUG zuul.Scheduler: Processing trigger event <TriggerEvent comment-added openstack-dev/sandbox master 201002,1 Verified:1> 2015-07-14 14:07:24,922 DEBUG zuul.Scheduler: Project openstack-dev/sandbox not found 2015-07-14 14:07:24,922 DEBUG zuul.Scheduler: Run handler sleeping My /etc/zuul/layout/layout.yaml looks like this: projects: - name: openstack-dev/sandbox check: - noop-check-communication My /etc/jenkins_jobs/config/projects.yaml looks like this: - project: name: sandbox github-org: openstack-dev node: master vendor: myvendor jobs: - noop-check-communication - dsvm-tempest-full: node: 'devstack_slave || devstack-precise-check || d-p-c' And Jenkins master works fine. Does anyone know what is going on here ? Thanks. __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<http://[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks & Regards, Abhishek Cloudbyte Inc.<http://www.cloudbyte.com> -- Thanks & Regards, Abhishek Cloudbyte Inc.<http://www.cloudbyte.com> __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<mailto:[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<http://[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks & Regards, Abhishek Cloudbyte Inc.<http://www.cloudbyte.com> __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<mailto:[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<http://[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- [https://docs.google.com/uc?export=download&id=0Byq0j7ZjFlFKV3ZCWnlMRXBCcU0&revid=0Byq0j7ZjFlFKa2V5VjdBSjIwUGx6bUROS2IrenNwc0kzd2IwPQ] Thanks & Regards, Abhishek Cloudbyte Inc.<http://www.cloudbyte.com> __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<mailto:[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<mailto:[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<mailto:[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<http://[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- [https://docs.google.com/uc?export=download&id=0Byq0j7ZjFlFKV3ZCWnlMRXBCcU0&revid=0Byq0j7ZjFlFKa2V5VjdBSjIwUGx6bUROS2IrenNwc0kzd2IwPQ] Thanks & Regards, Abhishek Cloudbyte Inc.<http://www.cloudbyte.com> __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<mailto:[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<http://[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- [https://docs.google.com/uc?export=download&id=0Byq0j7ZjFlFKV3ZCWnlMRXBCcU0&revid=0Byq0j7ZjFlFKa2V5VjdBSjIwUGx6bUROS2IrenNwc0kzd2IwPQ] Thanks & Regards, Abhishek Cloudbyte Inc.<http://www.cloudbyte.com> -- [https://docs.google.com/uc?export=download&id=0Byq0j7ZjFlFKV3ZCWnlMRXBCcU0&revid=0Byq0j7ZjFlFKa2V5VjdBSjIwUGx6bUROS2IrenNwc0kzd2IwPQ] Thanks & Regards, Abhishek Cloudbyte Inc.<http://www.cloudbyte.com> __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe<mailto:[email protected]?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
