Re: [openstack-dev] [designate] Designate performance issues
Hi Graham, thanks for your suggestion. But in fact the initial import was a simple while-curl scripts with no concurrency. With this script, a request will not be sent unless previous one gets reponse from designate-api. So I think it's not the rate of initial importing but the number of records that matters. I have filed this bug with detailed error logs here: https://bugs.launchpad.net/designate/+bug/1434479 On Thu, Mar 19, 2015 at 9:59 PM, Hayes, Graham graham.ha...@hp.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/19/2015 07:41 AM, stanzgy wrote: Hi all. I have setup kilo designate services with powerdns backend and mysql innodb storage in a single node. The services function well at first. However, after inserting 13k A records via API within 3 domains (5k, 5k, 3k for each), the service stops working. designate-api returns 500 and many RPCs timeout designate-central takes 100% cpu, seems trying hard updating domains but failed designate-mdns also takes 100% cpu, flooded with Including all tenants items in query results logs powerdns gets timeout errors during AXFR zones The server doesn't seem to turn any better after suffering in this state for hours. What I could do to recover the service is to cleanup databases and restart the service. My question is: 1. Is it not recommended to create too many records in a single domain? 2. Any suggestions to improve this situation? -- Best Regards, Zhang Gengyuan http://www.hp.com/ First off, for that initial burst of activity, I would disable debug level logging. How did you try and add them? Was it via a calling the API as fast as possible until it fell over? I would recommend rate limiting the initial import if it was. Do you have any logs from the services? Graham __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Zhang Gengyuan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [designate] Designate performance issues
Hi vinod, thanks for you reply. I have report a bug with related log snippets here: https://bugs.launchpad.net/designate/+bug/1434479 On Thu, Mar 19, 2015 at 10:11 PM, Vinod Mangalpally vinod.m...@rackspace.com wrote: Hi Zhang, Thank you for reporting the bug. The number of records does not seem too high. At this point I do not have a suggestion to improve the situation, but I will investigate this. Could you file a bug report? Relevant log snippets would also be helpful. --vinod From: stanzgy stan@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Thursday, March 19, 2015 2:39 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: [openstack-dev] [designate] Designate performance issues Hi all. I have setup kilo designate services with powerdns backend and mysql innodb storage in a single node. The services function well at first. However, after inserting 13k A records via API within 3 domains (5k, 5k, 3k for each), the service stops working. designate-api returns 500 and many RPCs timeout designate-central takes 100% cpu, seems trying hard updating domains but failed designate-mdns also takes 100% cpu, flooded with Including all tenants items in query results logs powerdns gets timeout errors during AXFR zones The server doesn't seem to turn any better after suffering in this state for hours. What I could do to recover the service is to cleanup databases and restart the service. My question is: 1. Is it not recommended to create too many records in a single domain? 2. Any suggestions to improve this situation? -- Best Regards, Zhang Gengyuan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Zhang Gengyuan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [designate] Designate performance issues
Hi all. I have setup kilo designate services with powerdns backend and mysql innodb storage in a single node. The services function well at first. However, after inserting 13k A records via API within 3 domains (5k, 5k, 3k for each), the service stops working. designate-api returns 500 and many RPCs timeout designate-central takes 100% cpu, seems trying hard updating domains but failed designate-mdns also takes 100% cpu, flooded with Including all tenants items in query results logs powerdns gets timeout errors during AXFR zones The server doesn't seem to turn any better after suffering in this state for hours. What I could do to recover the service is to cleanup databases and restart the service. My question is: 1. Is it not recommended to create too many records in a single domain? 2. Any suggestions to improve this situation? -- Best Regards, Zhang Gengyuan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra] Jenkins gate jobs fails
hi, Clark, I have tried recheck several times and got jenkins passed. Thanks for the explanation. On Fri, Jul 11, 2014 at 1:29 PM, Clark Boylan clark.boy...@gmail.com wrote: On Thu, Jul 10, 2014 at 10:12 PM, stanzgy stan@gmail.com wrote: Several jenkins gate jobs failed since some lib packages in ubuntu source are missing and devstack failed to setup the tempest env. Could there be someone help to fix this? I have filed the bug here: https://bugs.launchpad.net/openstack-ci/+bug/1340514 2014-07-11 02:35:42.333 | + apt_get install qemu-kvm 2014-07-11 02:35:42.336 | + sudo DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy= apt-get --option Dpkg::Options::=--force-confold --assume-yes install qemu-kvm 2014-07-11 02:35:42.358 | Reading package lists... 2014-07-11 02:35:42.602 | Building dependency tree... 2014-07-11 02:35:42.604 | Reading state information... 2014-07-11 02:35:42.769 | The following packages were automatically installed and are no longer required: 2014-07-11 02:35:42.769 | python-colorama python-distlib python-html5lib 2014-07-11 02:35:42.769 | Use 'apt-get autoremove' to remove them. 2014-07-11 02:35:42.796 | The following extra packages will be installed: 2014-07-11 02:35:42.796 | cpu-checker ipxe-qemu libbluetooth3 libbrlapi0.6 libcaca0 libfdt1 2014-07-11 02:35:42.796 | libsdl1.2debian libseccomp2 libspice-server1 libusbredirparser1 libxen-4.4 2014-07-11 02:35:42.796 | libxenstore3.0 libyajl2 msr-tools qemu-keymaps qemu-system-common 2014-07-11 02:35:42.796 | qemu-system-x86 seabios 2014-07-11 02:35:42.797 | Suggested packages: 2014-07-11 02:35:42.797 | samba vde2 sgabios 2014-07-11 02:35:42.798 | The following NEW packages will be installed: 2014-07-11 02:35:42.798 | cpu-checker ipxe-qemu libbluetooth3 libbrlapi0.6 libcaca0 libfdt1 2014-07-11 02:35:42.798 | libsdl1.2debian libseccomp2 libspice-server1 libusbredirparser1 libxen-4.4 2014-07-11 02:35:42.798 | libxenstore3.0 libyajl2 msr-tools qemu-keymaps qemu-kvm qemu-system-common 2014-07-11 02:35:42.798 | qemu-system-x86 seabios 2014-07-11 02:35:42.851 | 0 upgraded, 19 newly installed, 0 to remove and 0 not upgraded. 2014-07-11 02:35:42.851 | Need to get 291 kB/3985 kB of archives. 2014-07-11 02:35:42.851 | After this operation, 20.4 MB of additional disk space will be used. 2014-07-11 02:35:42.851 | Err http://mirror.rackspace.com/ubuntu/ trusty-security/main libxenstore3.0 amd64 4.4.0-0ubuntu5.1 2014-07-11 02:35:42.851 | 404 Not Found 2014-07-11 02:35:42.855 | Err http://mirror.rackspace.com/ubuntu/ trusty-security/main libxen-4.4 amd64 4.4.0-0ubuntu5.1 2014-07-11 02:35:42.855 | 404 Not Found 2014-07-11 02:35:42.858 | E: Failed to fetch http://mirror.rackspace.com/ubuntu/pool/main/x/xen/libxenstore3.0_4.4.0-0ubuntu5.1_amd64.deb 404 Not Found 2014-07-11 02:35:42.858 | 2014-07-11 02:35:42.858 | E: Failed to fetch http://mirror.rackspace.com/ubuntu/pool/main/x/xen/libxen-4.4_4.4.0-0ubuntu5.1_amd64.deb 404 Not Found 2014-07-11 02:35:42.858 | 2014-07-11 02:35:42.858 | E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? -- Best Regards ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev This bug is a duplicate of https://bugs.launchpad.net/bugs/1286818 I have gone ahead and marked it that way. The issue here is the rackspace mirrors periodically go sideways and don't work properly. We think this is because they are not syncing from ubuntu safely. There are a couple options to fix this. We can run our own ubuntu, centos, and fedora mirrors. There has been some work to get this going, https://review.openstack.org/#/c/89928/1 and https://review.openstack.org/#/c/90875/. pleia2 and dprince should know more. Rackspace could also correct their mirror syncing. Or we could possibly point a different mirrors entirely but that probably won't be any better than rackspace in the long run due to the Internet being unreliable. The first option gives us the most control and ability to react if things break. Feel free to review and/or update those changes as appropriate. Clark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Gengyuan Zhang NetEase Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Infra] Jenkins gate jobs fails
Several jenkins gate jobs failed since some lib packages in ubuntu source are missing and devstack failed to setup the tempest env. Could there be someone help to fix this? I have filed the bug here: https://bugs.launchpad.net/openstack-ci/+bug/1340514 2014-07-11 02:35:42.333 | + apt_get install qemu-kvm 2014-07-11 02:35:42.336 | + sudo DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy= apt-get --option Dpkg::Options::=--force-confold --assume-yes install qemu-kvm 2014-07-11 02:35:42.358 | Reading package lists... 2014-07-11 02:35:42.602 | Building dependency tree... 2014-07-11 02:35:42.604 | Reading state information... 2014-07-11 02:35:42.769 | The following packages were automatically installed and are no longer required: 2014-07-11 02:35:42.769 | python-colorama python-distlib python-html5lib 2014-07-11 02:35:42.769 | Use 'apt-get autoremove' to remove them. 2014-07-11 02:35:42.796 | The following extra packages will be installed: 2014-07-11 02:35:42.796 | cpu-checker ipxe-qemu libbluetooth3 libbrlapi0.6 libcaca0 libfdt1 2014-07-11 02:35:42.796 | libsdl1.2debian libseccomp2 libspice-server1 libusbredirparser1 libxen-4.4 2014-07-11 02:35:42.796 | libxenstore3.0 libyajl2 msr-tools qemu-keymaps qemu-system-common 2014-07-11 02:35:42.796 | qemu-system-x86 seabios 2014-07-11 02:35:42.797 | Suggested packages: 2014-07-11 02:35:42.797 | samba vde2 sgabios 2014-07-11 02:35:42.798 | The following NEW packages will be installed: 2014-07-11 02:35:42.798 | cpu-checker ipxe-qemu libbluetooth3 libbrlapi0.6 libcaca0 libfdt1 2014-07-11 02:35:42.798 | libsdl1.2debian libseccomp2 libspice-server1 libusbredirparser1 libxen-4.4 2014-07-11 02:35:42.798 | libxenstore3.0 libyajl2 msr-tools qemu-keymaps qemu-kvm qemu-system-common 2014-07-11 02:35:42.798 | qemu-system-x86 seabios 2014-07-11 02:35:42.851 | 0 upgraded, 19 newly installed, 0 to remove and 0 not upgraded. 2014-07-11 02:35:42.851 | Need to get 291 kB/3985 kB of archives. 2014-07-11 02:35:42.851 | After this operation, 20.4 MB of additional disk space will be used. 2014-07-11 02:35:42.851 | Err http://mirror.rackspace.com/ubuntu/ trusty-security/main libxenstore3.0 amd64 4.4.0-0ubuntu5.1 2014-07-11 02:35:42.851 | 404 Not Found 2014-07-11 02:35:42.855 | Err http://mirror.rackspace.com/ubuntu/ trusty-security/main libxen-4.4 amd64 4.4.0-0ubuntu5.1 2014-07-11 02:35:42.855 | 404 Not Found 2014-07-11 02:35:42.858 | E: Failed to fetch http://mirror.rackspace.com/ubuntu/pool/main/x/xen/libxenstore3.0_4.4.0-0ubuntu5.1_amd64.deb 404 Not Found 2014-07-11 02:35:42.858 | 2014-07-11 02:35:42.858 | E: Failed to fetch http://mirror.rackspace.com/ubuntu/pool/main/x/xen/libxen-4.4_4.4.0-0ubuntu5.1_amd64.deb 404 Not Found 2014-07-11 02:35:42.858 | 2014-07-11 02:35:42.858 | E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? -- Best Regards ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] is tenant-id in network API?
This patch should fix the problem, sorry for the mistakes above.
On Mon, Jul 7, 2014 at 1:46 PM, Andreas Jaeger a...@suse.com wrote:
On 07/07/2014 05:56 AM, Anne Gentle wrote:
Please check to see if this patch fixes the issue:
https://review.openstack.org/1050
It should be:
https://review.openstack.org/105060
Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Best Regards,
Gengyuan Zhang
NetEase Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] is tenant-id in network API?
hi, I use the newest api-ref master branch to generate the neutron API docs
and
there is no tenant_id as part of the url.
I think it is because the api-ref website hasn't updated with the newest
api-ref repo.
On Mon, Jul 7, 2014 at 10:53 AM, Anne Gentle a...@openstack.org wrote:
We have an API doc bug to track this issue:
https://bugs.launchpad.net/openstack-api-site/+bug/1334837
Thanks for reporting, have marked it High.
Anne
On Sun, Jul 6, 2014 at 9:41 PM, Damon Wang damon.dev...@gmail.com wrote:
Agree,
http://docs.openstack.org/api/openstack-network/2.0/content/networks.html
and http://developer.openstack.org/api-ref-networking-v2.html both have
this problem.
2014-07-07 10:25 GMT+08:00 Yongsheng Gong gong...@unitedstack.com:
Hi,
Today, I found the {tenant_id} is written in neutron API URLs at
http://developer.openstack.org/api-ref-networking-v2.html.
But when I tried to access it accordingly, it failed.
I don't think we have tenant_id in network API URL.
any Idea?
regards,
yong sheng gong
UnitedStack Inc.
___
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openst...@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Best Regards,
Gengyuan Zhang
NetEase Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron]One security issue about floating ip
I have filed this bug on nova https://bugs.launchpad.net/nova/+bug/1334938 On Fri, Jun 27, 2014 at 10:19 AM, Yongsheng Gong gong...@unitedstack.com wrote: I have reported it on neutron project https://bugs.launchpad.net/neutron/+bug/1334926 On Fri, Jun 27, 2014 at 5:07 AM, Vishvananda Ishaya vishvana...@gmail.com wrote: I missed that going in, but it appears that clean_conntrack is not done on disassociate, just during migration. It sounds like we should remove the explicit call in migrate, and just always call it from remove_floating_ip. Vish On Jun 26, 2014, at 1:48 PM, Brian Haley brian.ha...@hp.com wrote: Signed PGP part I believe nova-network does this by using 'conntrack -D -r $fixed_ip' when the floating IP goes away (search for clean_conntrack), Neutron doesn't when it removes the floating IP. Seems like it's possible to close most of that gap in the l3-agent - when it removes the IP from it's qg- interface it can do a similar operation. -Brian On 06/26/2014 03:36 PM, Vishvananda Ishaya wrote: I believe this will affect nova-network as well. We probably should use something like the linux cutter utility to kill any ongoing connections after we remove the nat rule. Vish On Jun 25, 2014, at 8:18 PM, Xurong Yang ido...@gmail.com wrote: Hi folks, After we create an SSH connection to a VM via its floating ip, even though we have removed the floating ip association, we can still access the VM via that connection. Namely, SSH is not disconnected when the floating ip is not valid. Any good solution about this security issue? Thanks Xurong Yang ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Gengyuan Zhang NetEase Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
