[openstack-dev] [keystone] Need info on the correct location to place the certificates / correct tags to specify the same
Hi All, While upgrading from Kilo to Liberty, I am seeing these warnings in the logs:- ./keystone/keystone.log:2016-05-11 13:40:34.013 20402 WARNING oslo_config.cfg [-] Option "certfile" from group "ssl" is deprecated. Use option "certfile" from group "eventlet_server_ssl". ./keystone/keystone.log:2016-05-11 13:40:34.013 20402 WARNING oslo_config.cfg [-] Option "certfile" from group "eventlet_server_ssl" is deprecated for removal. Its value may be silently ignored in the future. ./keystone/keystone.log:2016-05-11 13:40:34.013 20402 WARNING oslo_config.cfg [-] Option "keyfile" from group "ssl" is deprecated. Use option "keyfile" from group "eventlet_server_ssl". ./keystone/keystone.log:2016-05-11 13:40:34.013 20402 WARNING oslo_config.cfg [-] Option "keyfile" from group "eventlet_server_ssl" is deprecated for removal. Its value may be silently ignored in the future. ./keystone/keystone.log:2016-05-11 13:40:34.013 20402 WARNING oslo_config.cfg [-] Option "ca_certs" from group "ssl" is deprecated. Use option "ca_certs" from group "eventlet_server_ssl". ./keystone/keystone.log:2016-05-11 13:40:34.013 20402 WARNING oslo_config.cfg [-] Option "ca_certs" from group "eventlet_server_ssl" is deprecated for removal. Its value may be silently ignored in the future. It looks like the parameters certfile, keyfile, ca_certs are going to be deprecated(might be deprecated by now) in future releases. For running keystone with TLS, I need to specify the location of my certificates in some configuration file. Does the above logs mean that we are going to store the certs in some standard/default directory? I tried to find any documentation specifying these changes or any configuration updates needed to support these changes, but couldn't find any. Can someone please help me out in identifying where the right configuration should be? Thanks. *Rahul Sharma* *MS in Computer Science, 2016* College of Computer and Information Science, Northeastern University Mobile: 801-706-7860 Email: rahulsharma...@gmail.com Linkedin: www.linkedin.com/in/rahulsharmaait __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] how to recover instances with different hypervisor-id on the same compute node
Hi All, Due to a hostname change, we ended up having a new hypervisor-id for one of our compute nodes. It was already running two instances and then, we didn't catch the updated hypervisor-id. This ended up with new instances spawning up with new hypervisor-id. Now, the instances with older hypervisor-id are not seen in "virsh list", rebooting those instances keep them rebooting. [root (admin)]# nova hypervisor-servers compute-90 +--+---+---+-+ | ID | Name | Hypervisor ID | Hypervisor Hostname | +--+---+---+-+ | 9688dcef-2836-496f-8b70-099638b73096 | instance-0712 | 40 | compute-90.test.edu | | f7373cd6-96a0-4643-9137-732ea5353e94 | instance-0b74 | 40 | compute-90.test.edu | | c8926585-a260-45cd-b008-71df2124b364 | instance-1270 | 92 | compute-90.test.edu | | a0aa3f5f-d49b-43a6-8465-e7865bb68d57 | instance-18de | 92 | compute-90.test.edu | | d729f9f4-fcae-4abe-803c-e9474e533a3b | instance-16e0 | 92 | compute-90.test.edu | | 30a6a05d-a170-4105-9987-07a875152907 | instance-17e4 | 92 | compute-90.test.edu | | 6e0fa25b-569d-4e9e-b57d-4c182c1c23ea | instance-18f8 | 92 | compute-90.test.edu | | 5964f6cc-eec3-493a-81fe-7fb616c89a8f | instance-18fa | 92 | compute-90.test.edu | +--+---+---+-+ [root@compute-90]# virsh list IdName State 112 instance-1270 running 207 instance-18de running 178 instance-16e0 running 189 instance-17e4 running 325 instance-18f8 running 336 instance-18fa running Instances not visible: instance-0712 and instance-0b74 Is there a way to recover from this step? I can delete the old services(nova service-delete ), but I am confused whether it will lead to loss of already running instances with old hypervisor-id? Is there a way I can update the state of those instances to use hypervisor-id as 92 instead of 40? Kindly do let me know if you have any suggestions. Thanks. *Rahul Sharma* *MS in Computer Science, 2016* College of Computer and Information Science, Northeastern University Mobile: 801-706-7860 Email: rahulsharma...@gmail.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-operators] [neutron] openvswitch-agent spins up too many /bin/ovsdb-client processes
Thanks Hrushi. After further troubleshooting, we found that somehow openvswitch-agent was reading a file named "sensu" from /etc/sudoers.d directory and was failing in reading it, and we haven't configured anything like that in neutron configs. Removing that helped in bringing everything back to normal, but still its not clear why it was reading that file. We are trying to figure that out. *Rahul Sharma* *MS in Computer Science, 2016* College of Computer and Information Science, Northeastern University Mobile: 801-706-7860 Email: rahulsharma...@gmail.com On Mon, Mar 14, 2016 at 9:10 PM, Gangur, Hrushikesh < hrushikesh.gan...@hpe.com> wrote: > Rahul – it seems your issue is similar to the one reported here, probably > due to hostname resolution issue. > > https://bugs.launchpad.net/charms/+source/quantum-gateway/+bug/1405588 > > > > Regards~hrushi > > > > *From:* Rahul Sharma [mailto:rahulsharma...@gmail.com > <rahulsharma...@gmail.com>] > *Sent:* Monday, March 14, 2016 3:32 PM > *To:* openstack <openst...@lists.openstack.org>; OpenStack Development > Mailing List <openstack-dev@lists.openstack.org>; > openstack-operat...@lists.openstack.org > *Subject:* [Openstack-operators] [neutron] openvswitch-agent spins up too > many /bin/ovsdb-client processes > > > > Hi All, > > > > We are trying to debug an issue with our production environment. We are > seeing neutron-openvswitch-agent starts failing after some time (1-2 days). > After debugging, we found that there are large number of entries for the > ovsdb-client. On some nodes, it crosses more than 330 processes and then > ovsdb process starts failing. > > 1. root 30689 1 0 00:37 ?00:00:00 /bin/ovsdb-client > monitor Interface name,ofport --format=json > > 2. root 30804 1 0 00:38 ?00:00:00 /bin/ovsdb-client > monitor Interface name,ofport --format=json > > 3. root 30909 1 0 00:38 ?00:00:00 /bin/ovsdb-client > monitor Interface name,ofport --format=json > > > > Pastebin link for the processes: http://pastebin.com/QGQC0Jrt > > Pastebin link with openvswitch starting all of them: > http://pastebin.com/repHMkHu > > > > In logs, we start getting errors as:- > > Mar 14 05:41:29 node2 ovs-vsctl: ovs|1|fatal_signal|WARN|terminating > with signal 14 (Alarm clock) > > Mar 14 05:41:39 node2 ovs-vsctl: ovs|1|fatal_signal|WARN|terminating > with signal 14 (Alarm clock) > > Mar 14 05:41:49 node2 ovs-vsctl: ovs|1|fatal_signal|WARN|terminating > with signal 14 (Alarm clock) > > Mar 14 05:49:30 node2 ovs-vsctl: > ovs|1|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection > failed (Protocol error) > > Mar 14 05:49:32 node2 ovs-vsctl: > ovs|1|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection > failed (Protocol error) > > Mar 14 05:49:34 node2 ovs-vsctl: > ovs|1|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection > failed (Protocol error) > > > > Openvswitch version:- > > [root@node2 ~(openstack_admin)]# ovs-vsctl --version > > ovs-vsctl (Open vSwitch) 2.4.0 > > Compiled Sep 4 2015 09:49:34 > > DB Schema 7.12.1 > > > > We have to restart openvswitch service everytime and that clears up all > the processes. We are trying to figure out why so many processes are > getting started by neutron-agent? Also, we found that if we restart the > host's networking, one new process for the /bin/ovsdb-client starts. We > checked and found that we don't have any network fluctuations or any > nic-flappings. Are there any pointers where we should be looking into? It > occurs on both controller and compute nodes. > > > *Rahul Sharma* > *MS in Computer Science, 2016* > College of Computer and Information Science, Northeastern University > Mobile: 801-706-7860 > Email: rahulsharma...@gmail.com > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] openvswitch-agent spins up too many /bin/ovsdb-client processes
Hi All, We are trying to debug an issue with our production environment. We are seeing neutron-openvswitch-agent starts failing after some time (1-2 days). After debugging, we found that there are large number of entries for the ovsdb-client. On some nodes, it crosses more than 330 processes and then ovsdb process starts failing. 1. root 30689 1 0 00:37 ?00:00:00 /bin/ovsdb-client monitor Interface name,ofport --format=json 2. root 30804 1 0 00:38 ?00:00:00 /bin/ovsdb-client monitor Interface name,ofport --format=json 3. root 30909 1 0 00:38 ?00:00:00 /bin/ovsdb-client monitor Interface name,ofport --format=json Pastebin link for the processes: http://pastebin.com/QGQC0Jrt Pastebin link with openvswitch starting all of them: http://pastebin.com/repHMkHu In logs, we start getting errors as:- Mar 14 05:41:29 node2 ovs-vsctl: ovs|1|fatal_signal|WARN|terminating with signal 14 (Alarm clock) Mar 14 05:41:39 node2 ovs-vsctl: ovs|1|fatal_signal|WARN|terminating with signal 14 (Alarm clock) Mar 14 05:41:49 node2 ovs-vsctl: ovs|1|fatal_signal|WARN|terminating with signal 14 (Alarm clock) Mar 14 05:49:30 node2 ovs-vsctl: ovs|1|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (Protocol error) Mar 14 05:49:32 node2 ovs-vsctl: ovs|1|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (Protocol error) Mar 14 05:49:34 node2 ovs-vsctl: ovs|1|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (Protocol error) Openvswitch version:- [root@node2 ~(openstack_admin)]# ovs-vsctl --version ovs-vsctl (Open vSwitch) 2.4.0 Compiled Sep 4 2015 09:49:34 DB Schema 7.12.1 We have to restart openvswitch service everytime and that clears up all the processes. We are trying to figure out why so many processes are getting started by neutron-agent? Also, we found that if we restart the host's networking, one new process for the /bin/ovsdb-client starts. We checked and found that we don't have any network fluctuations or any nic-flappings. Are there any pointers where we should be looking into? It occurs on both controller and compute nodes. *Rahul Sharma* *MS in Computer Science, 2016* College of Computer and Information Science, Northeastern University Mobile: 801-706-7860 Email: rahulsharma...@gmail.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [puppet] need help understanding specific puppet logs in openstack installation
Hi All, I am trying the RDO installation of Openstack using puppet. It works fine when I puppetize my controller node for the first time after fresh installation of OS. There are no errors thrown at terminal during the installation. However, if I try to rerun the puppet agent, it configures something additional this time which is not configured during the first run. Mentioned below are the logs it prints to terminal:- Notice: /File[/etc/sysconfig/iptables]/seluser: seluser changed 'unconfined_u' to 'system_u' Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[ovs-cleanup-service]/enable: enable changed 'false' to 'true' Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron::Plugins::Ovs::Bridge[physext1:br-ex]/Vs_bridge[br-ex]/external_ids: external_ids changed '' to 'bridge-id=br-ex' Notice: /Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns: executed successfully Notice: Finished catalog run in 36.61 seconds >From these logs, I am unable to figure out which of the parameters were not configured when it was run for the first time so that I can try fixing them in the script itself. I want to avoid the dependency of running puppet multiple times. Looking at the *.conf files on the controller node doesn't help much. Can someone please help me out in understanding what the above mentioned logs point to? Any pointers would be really helpful. Thanks. *Rahul Sharma* *MS in Computer Science, 2016* College of Computer and Information Science, Northeastern University Mobile: 801-706-7860 Email: rahulsharma...@gmail.com Linkedin: www.linkedin.com/in/rahulsharmaait __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Nova] understanding SSL behavior
Hi All, I am trying to configure the endpoints to communicate over https. I am trying to debug a particular behavior of code but unable to relate my sequence of actions with the behavior of code. Kindly do guide me to understand the below mentioned scenario. For SSL, I have generated a self-signed CA cert and used it for signing CSR request of host/controller. I placed the CA cert in the trusted-root authority of my host and all the services work fine. They are able to talk with each other over https. I was able to access the url https://:8774 from anywhere. I went ahead and modified the nova.conf and added ssl_ca_file in [DEFAULT] section. [DEFAULT] ... ssl_ca_file= ssl_cert_file= ssl_key_file= ... Nova services come up fine, but now I am unable to access the url https://:8774. If I again remove the ssl_ca_file from nova.conf, it again starts working fine. Looking at the code, I could see that its getting used in nova/wsgi.py. if CONF.ssl_ca_file: ssl_kwargs['ca_certs'] = ca_file ssl_kwargs['cert_reqs'] = ssl.CERT_REQUIRED I am missing some very basic thing here, can someone please help me to understand the sequence of steps going on and what do I need to do to communicate with the service. The service is running and listening on port 8774, but it looks like I might have to provide something else with the request to communicate with the service. Since various other services would be communicating with nova, do I need to configure some specific parameter in those services? Any pointers would be really helpful. Thanks. *Rahul Sharma* *MS in Computer Science, 2016* College of Computer and Information Science, Northeastern University Mobile: 801-706-7860 Email: rahulsharma...@gmail.com Linkedin: www.linkedin.com/in/rahulsharmaait __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] Instances running on VMware ESXi are unable to configure IP
Hi Ramon, We need to add flows to br-tun bridge on compute node. Kindly note that since in our design, we have used eth2 attached to br-int of ESX, added br-int in promiscuous mode, so there is no segregation of vm's based on tenant for ESX host. When we add eth2 to br-int of OVS, we assign a tag to that port. root@nova-compute:~# ovs-vsctl add-port br-int eth2 tag=1 You can now check what is the port-id associated to eth2 in OVS's br-int by using the command:- root@nova-compute:~# ovs-dpctl show br-int Note down the port-number associated to eth2 since it would be used while adding flow-rules to br-tun. On br-tun, we need to add rules for outgoing packet as well as for incoming packet. For outgoing packet, we will add rule to encapsulate the packet in GRE tunnel header. Similarly, for incoming packet, we will add rule to remove the GRE header. Listed below are the rules which we added:- root@nova-compute:~# ovs-ofctl add-flow br-tun priority=4,in_port=1,dl_vlan=1,actions=set_tunnel:0x1,NORMAL root@nova-compute:~# ovs-ofctl add-flow br-tun priority=3,tun_id=0x1,actions=mod_vlan_vid:1,NORMAL Here, in_port is the port-number of eth2 on br-int. Actions represents what is to be done if the packet matches with that flow. If you face any issues, you can do tcpdump on each interface/switch starting from eth2 - br-int - br-tun - eth1 and similarly on network node. You can dump the flows of OVS using commands ovs-ofctl dump-flows bridg-name and can then check the packet_counters associatef with those flows to see which flow is getting hit. If you are entering any wrong rules, you can debug them using this. Hope it will help you. Thanks and Regards Rahul Sharma ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] Instances running on VMware ESXi are unable to configure IP
On Wed, Sep 18, 2013 at 9:09 PM, Ramon Acedo ramon.ac...@ubuntu.com wrote: Hi Rahul, On 12 September 2013 18:58, Rahul Sharma rahulsharma...@gmail.com wrote: Today we were able to achieve the end-to-end flow of traffic by adding rules manually to the openvswitch-switches in nova-compute vm. If support for the configuring flows in switches is added through API's, maybe we can support openvswitch as well. Though, ideally one should not use vSwitch as its having minimal capabilities and one should always go with the DVS for ESX. Would you mind sharing the rules you used for Open vSwitch to allow the traffic flow through the Nova Compute VM? Using vSphere DVS or vSphere standard vSwitch shouldn't make any difference for this setup (it does in the vSphere network design though). Hmm, I am not sure whether vSphere DVS performs intelligent MAC learning which vSwitch lacks, but if it does, then we can use port-group br-int without adding it in promiscuous mode. Ideally, nova-compute vm should not be dependent on its placement with respect to the physical compute node, so the design should be that all the traffic of instances should be sent out using br-int to physical switch, and then should be forwarded to eth2 of compute-vm. Also, segregation based on tenant can be achieved by creating port-groups of different vlan-id for each tenant and attaching those port-groups to instances, rather than the current br-int of vswitch. If all these things are taken care, and flows are added to OVS, then we can achieve quantum-networking using OVS in vSphere similar to the the one in KVM. In the network plugin compatibility matrix with compute drivers [1] it says that only Nicira NVP and Plumgrid are supported with VMware ESXi as Nova Compute, which is different from the vSphere DVS (Distributed vSwitch). However, I believe that Open vSwitch has everything it's needed to provide the same level of functionality with ESXi/vCenter than with KVM OpenStack setups (including per tenant network isolation). Yes, I agree to that. Only thing which is missing is the proper design to achieve the same on ESXi/vCenter. Thanks! [1] http://docs.openstack.org/trunk/openstack-network/admin/content/flexibility.html -Regards Rahul Sharma ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] Patching Horizon Code when installed using apt-get install
Hi Shake, As per the patch, it seems that they have added check for v2_0. def tenant_create(request, name, description=None, enabled=None, domain=None): manager = VERSIONS.get_project_manager(request, admin=True) *if VERSIONS.active 3:* return manager.create(name, description, enabled) else: return manager.create(name, domain, description=description, enabled=enabled) Looking at keystoneclient code of grizzly release, I could see that in v2_0 directory, users.py has the method update_own_password implemented. Looks like problem is not because of this version difference. -Regards Rahul Sharma On Tue, Jun 25, 2013 at 6:06 PM, Shake Chen shake.c...@gmail.com wrote: the patch use keystone v3 , in grizzly horizon use keystone v2. On Tue, Jun 25, 2013 at 4:52 PM, Rahul Sharma rahulsharma...@gmail.comwrote: Hi All, I have setup multi-node openstack setup using grizzly release and ubuntu 12.04 distribution. Since there is no support for individual user to change his/her password, someone has provided a patch for the same. Ref:- https://review.openstack.org/#/c/23901/31 Now I am trying to apply the patch by copying the files to /usr/share/openstack-dashboard directory. On restarting the Apache service, I am getting an error. I have attached the logs with the mail. Even if I revert back the patch, restart apache service and try to access Horizon, still I get the same error. I have to remove horizon, remove /usr/share/openstack-dashboard directory and reinstall horizon to get rid of this error. I might be missing some step inbetween to apply this patch to the code. Can someone help me out in identifying how to apply this patch to the current installation. I couldn't find any documentation listing how to apply such patches other than for git repositories. Thanks and Regards Rahul Sharma ___ Mailing list: https://launchpad.net/~openstack Post to : openst...@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Shake Chen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
