Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-28 Thread Kevin Benton
What do you see in the Neutron server logs while it's not responding? On Tue, Feb 28, 2017 at 1:27 AM, Satyanarayana Patibandla wrote: > Hi Kevin, > > Thanks for your suggestion. I will modify the parameter value and will test > the changes. > > Could you please provide your suggestion on recover

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-28 Thread Satyanarayana Patibandla
Hi Kevin, Thanks for your suggestion. I will modify the parameter value and will test the changes. Could you please provide your suggestion on recovering to normal state after getting this error. Once we get this error the neutron CLI gives "504 gateway timeout". We tried to restart all neutron-s

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-28 Thread Kevin Benton
That particular update query is issued by the agent state report handler. And it looks like they might be falling behind based on the timestamp it's trying to update in the DB (14:46:35) and the log statement (14:50:29). Can you try increasing the rpc_state_report_workers value? If you haven't mod

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-27 Thread Satyanarayana Patibandla
Hi Kevin, After increasing the parameter values mentioned in the below mail, we are able to create few hundreds of VMs properly. There were no errors related to neutron. Our environment contain multiple regions. One of our team member by mistake ran all openstack service tempest tests against the

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-26 Thread Satyanarayana Patibandla
Hi, We increased api_workers,rpc_workers and metadata_workers based on the number of cores we are running on controller node ( the workers are half of the number of cores. i.e if we have 24 cores then we are running 12 workers for each). Increased rpc_connect_timeout to 180 and rpc_response_timeou

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-26 Thread Kevin Benton
Thanks for following up. Would you mind sharing the parameters you had to tune (db pool limits, etc) just in case someone comes across this same thread in a google search? Thanks, Kevin Benton On Sun, Feb 26, 2017 at 8:48 PM, Satyanarayana Patibandla < satya.patiban...@gmail.com> wrote: > Hi Sav

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-26 Thread Satyanarayana Patibandla
Hi Saverio, The issue seems to be related to neutron tuning. We observed the same issue with stable/ocata branch code. When we tuned few neutron parameters it is working fine. Thanks for your suggestion. Thanks, Satya.P On Wed, Feb 22, 2017 at 10:10 AM, Satyanarayana Patibandla < satya.patiban..

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-21 Thread Satyanarayana Patibandla
Hi Saverio, Thanks for your inputs. Will test with statable/ocata branch code and will share the result. Thanks, Satya.P On Wed, Feb 22, 2017 at 1:54 AM, Saverio Proto wrote: > Hello, > > I would use at least the stable/ocata branch. If you just use master > that is not supposed to be stable,

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-21 Thread Saverio Proto
Hello, I would use at least the stable/ocata branch. If you just use master that is not supposed to be stable, and also I am not sure if you can fill a bug against a specific commit in master. Saverio 2017-02-21 21:12 GMT+01:00 Satyanarayana Patibandla : > Hi Saverio, > > We have tried to create

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-21 Thread Satyanarayana Patibandla
Hi Saverio, We have tried to create 20 VMs each time using heat template. There is 1 sec time gap between each VM creation request. When we reached 114 VMs we got the error mentioned in the below mail.Heat template will boot instance from volume and it assigns floating IP to the instance. Except

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-21 Thread Saverio Proto
Hello Satya, I would fill a bug on launchpad for this issue. 114 VMs is not much. Can you identify how to trigger the issue to reproduce it ? or it just happens randomly ? When you say rebooting the network node, do you mean the server running the neutron-server process ? what version and distri

[Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

2017-02-21 Thread Satyanarayana Patibandla
Hi All, We are trying to deploy Openstack in our production environment. For networking we are using DVR with out L3 HA. We are able to create 114 VMs with out any issue. After creating 114 VMs we are getting the below error. Error: 504 Gateway Time-out The server didn't respond in time. Neutro