Re: [openstack-dev] [designate] Designate performance issues
Hi Graham, thanks for your suggestion. But in fact the initial import was a simple while-curl scripts with no concurrency. With this script, a request will not be sent unless previous one gets reponse from designate-api. So I think it's not the rate of initial importing but the number of records that matters. I have filed this bug with detailed error logs here: https://bugs.launchpad.net/designate/+bug/1434479 On Thu, Mar 19, 2015 at 9:59 PM, Hayes, Graham graham.ha...@hp.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/19/2015 07:41 AM, stanzgy wrote: Hi all. I have setup kilo designate services with powerdns backend and mysql innodb storage in a single node. The services function well at first. However, after inserting 13k A records via API within 3 domains (5k, 5k, 3k for each), the service stops working. designate-api returns 500 and many RPCs timeout designate-central takes 100% cpu, seems trying hard updating domains but failed designate-mdns also takes 100% cpu, flooded with Including all tenants items in query results logs powerdns gets timeout errors during AXFR zones The server doesn't seem to turn any better after suffering in this state for hours. What I could do to recover the service is to cleanup databases and restart the service. My question is: 1. Is it not recommended to create too many records in a single domain? 2. Any suggestions to improve this situation? -- Best Regards, Zhang Gengyuan http://www.hp.com/ First off, for that initial burst of activity, I would disable debug level logging. How did you try and add them? Was it via a calling the API as fast as possible until it fell over? I would recommend rate limiting the initial import if it was. Do you have any logs from the services? Graham __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Zhang Gengyuan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [designate] Designate performance issues
Hi vinod, thanks for you reply. I have report a bug with related log snippets here: https://bugs.launchpad.net/designate/+bug/1434479 On Thu, Mar 19, 2015 at 10:11 PM, Vinod Mangalpally vinod.m...@rackspace.com wrote: Hi Zhang, Thank you for reporting the bug. The number of records does not seem too high. At this point I do not have a suggestion to improve the situation, but I will investigate this. Could you file a bug report? Relevant log snippets would also be helpful. --vinod From: stanzgy stan@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Thursday, March 19, 2015 2:39 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: [openstack-dev] [designate] Designate performance issues Hi all. I have setup kilo designate services with powerdns backend and mysql innodb storage in a single node. The services function well at first. However, after inserting 13k A records via API within 3 domains (5k, 5k, 3k for each), the service stops working. designate-api returns 500 and many RPCs timeout designate-central takes 100% cpu, seems trying hard updating domains but failed designate-mdns also takes 100% cpu, flooded with Including all tenants items in query results logs powerdns gets timeout errors during AXFR zones The server doesn't seem to turn any better after suffering in this state for hours. What I could do to recover the service is to cleanup databases and restart the service. My question is: 1. Is it not recommended to create too many records in a single domain? 2. Any suggestions to improve this situation? -- Best Regards, Zhang Gengyuan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Zhang Gengyuan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [designate] Designate performance issues
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/19/2015 07:41 AM, stanzgy wrote: Hi all. I have setup kilo designate services with powerdns backend and mysql innodb storage in a single node. The services function well at first. However, after inserting 13k A records via API within 3 domains (5k, 5k, 3k for each), the service stops working. designate-api returns 500 and many RPCs timeout designate-central takes 100% cpu, seems trying hard updating domains but failed designate-mdns also takes 100% cpu, flooded with Including all tenants items in query results logs powerdns gets timeout errors during AXFR zones The server doesn't seem to turn any better after suffering in this state for hours. What I could do to recover the service is to cleanup databases and restart the service. My question is: 1. Is it not recommended to create too many records in a single domain? 2. Any suggestions to improve this situation? -- Best Regards, Zhang Gengyuan http://www.hp.com/ First off, for that initial burst of activity, I would disable debug level logging. How did you try and add them? Was it via a calling the API as fast as possible until it fell over? I would recommend rate limiting the initial import if it was. Do you have any logs from the services? Graham __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [designate] Designate performance issues
Hi Zhang, Thank you for reporting the bug. The number of records does not seem too high. At this point I do not have a suggestion to improve the situation, but I will investigate this. Could you file a bug report? Relevant log snippets would also be helpful. --vinod From: stanzgy stan@gmail.commailto:stan@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Thursday, March 19, 2015 2:39 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [designate] Designate performance issues Hi all. I have setup kilo designate services with powerdns backend and mysql innodb storage in a single node. The services function well at first. However, after inserting 13k A records via API within 3 domains (5k, 5k, 3k for each), the service stops working. designate-api returns 500 and many RPCs timeout designate-central takes 100% cpu, seems trying hard updating domains but failed designate-mdns also takes 100% cpu, flooded with Including all tenants items in query results logs powerdns gets timeout errors during AXFR zones The server doesn't seem to turn any better after suffering in this state for hours. What I could do to recover the service is to cleanup databases and restart the service. My question is: 1. Is it not recommended to create too many records in a single domain? 2. Any suggestions to improve this situation? -- Best Regards, Zhang Gengyuan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
