Re: [openstack-dev] [designate] Designate performance issues
Hi Graham, thanks for your suggestion. But in fact the initial import was a simple while-curl scripts with no concurrency. With this script, a request will not be sent unless previous one gets reponse from designate-api. So I think it's not the rate of initial importing but the number of records that matters. I have filed this bug with detailed error logs here: https://bugs.launchpad.net/designate/+bug/1434479 On Thu, Mar 19, 2015 at 9:59 PM, Hayes, Graham wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 03/19/2015 07:41 AM, stanzgy wrote: > > Hi all. I have setup kilo designate services with powerdns backend and > mysql innodb storage in a > single node. > > The services function well at first. However, after inserting 13k A > records via API within 3 domains (5k, 5k, 3k for each), the service > stops working. > > > > designate-api returns 500 and many RPCs timeout > > designate-central takes 100% cpu, seems trying hard updating domains > but failed > > designate-mdns also takes 100% cpu, flooded with "Including all > tenants items in query results" logs > > powerdns gets timeout errors during AXFR zones > > > > The server doesn't seem to turn any better after suffering in this > state for hours. What I could do to recover the service is to cleanup > databases and restart the service. > > > > My question is: > > 1. Is it not recommended to create too many records in a single domain? > > 2. Any suggestions to improve this situation? > > > > -- > > Best Regards, > > > > Zhang Gengyuan > <http://www.hp.com/> > > First off, for that initial burst of activity, I would disable debug > level logging. > > How did you try and add them? Was it via a calling the API as fast as > possible until it fell over? > I would recommend rate limiting the initial import if it was. > > Do you have any logs from the services? > > Graham > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best Regards, Zhang Gengyuan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [designate] Designate performance issues
Hi vinod, thanks for you reply. I have report a bug with related log snippets here: https://bugs.launchpad.net/designate/+bug/1434479 On Thu, Mar 19, 2015 at 10:11 PM, Vinod Mangalpally < vinod.m...@rackspace.com> wrote: > Hi Zhang, > > Thank you for reporting the bug. The number of records does not seem too > high. At this point I do not have a suggestion to improve the situation, > but I will investigate this. Could you file a bug report? Relevant log > snippets would also be helpful. > > --vinod > > From: stanzgy > Reply-To: "OpenStack Development Mailing List (not for usage questions)" < > openstack-dev@lists.openstack.org> > Date: Thursday, March 19, 2015 2:39 AM > To: "OpenStack Development Mailing List (not for usage questions)" < > openstack-dev@lists.openstack.org> > Subject: [openstack-dev] [designate] Designate performance issues > > Hi all. I have setup kilo designate services with powerdns backend > and mysql innodb storage in a single node. > The services function well at first. However, after inserting 13k A > records via API within 3 domains (5k, 5k, 3k for each), the service stops > working. > > designate-api returns 500 and many RPCs timeout > designate-central takes 100% cpu, seems trying hard updating domains but > failed > designate-mdns also takes 100% cpu, flooded with "Including all tenants > items in query results" logs > powerdns gets timeout errors during AXFR zones > > The server doesn't seem to turn any better after suffering in this state > for hours. What I could do to recover the service is to cleanup databases > and restart the service. > > My question is: > 1. Is it not recommended to create too many records in a single domain? > 2. Any suggestions to improve this situation? > > -- > Best Regards, > > Zhang Gengyuan > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Best Regards, Zhang Gengyuan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [designate] Designate performance issues
Hi all. I have setup kilo designate services with powerdns backend and mysql innodb storage in a single node. The services function well at first. However, after inserting 13k A records via API within 3 domains (5k, 5k, 3k for each), the service stops working. designate-api returns 500 and many RPCs timeout designate-central takes 100% cpu, seems trying hard updating domains but failed designate-mdns also takes 100% cpu, flooded with "Including all tenants items in query results" logs powerdns gets timeout errors during AXFR zones The server doesn't seem to turn any better after suffering in this state for hours. What I could do to recover the service is to cleanup databases and restart the service. My question is: 1. Is it not recommended to create too many records in a single domain? 2. Any suggestions to improve this situation? -- Best Regards, Zhang Gengyuan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra] Jenkins gate jobs fails
hi, Clark, I have tried recheck several times and got jenkins passed. Thanks for the explanation. On Fri, Jul 11, 2014 at 1:29 PM, Clark Boylan wrote: > On Thu, Jul 10, 2014 at 10:12 PM, stanzgy wrote: > > Several jenkins gate jobs failed since some lib packages in ubuntu source > > are missing and devstack failed to setup the tempest env. Could there be > > someone help to fix this? > > I have filed the bug here: > > https://bugs.launchpad.net/openstack-ci/+bug/1340514 > > > > > > 2014-07-11 02:35:42.333 | + apt_get install qemu-kvm > > 2014-07-11 02:35:42.336 | + sudo DEBIAN_FRONTEND=noninteractive > http_proxy= > > https_proxy= no_proxy= apt-get --option Dpkg::Options::=--force-confold > > --assume-yes install qemu-kvm > > 2014-07-11 02:35:42.358 | Reading package lists... > > 2014-07-11 02:35:42.602 | Building dependency tree... > > 2014-07-11 02:35:42.604 | Reading state information... > > 2014-07-11 02:35:42.769 | The following packages were automatically > > installed and are no longer required: > > 2014-07-11 02:35:42.769 | python-colorama python-distlib > python-html5lib > > 2014-07-11 02:35:42.769 | Use 'apt-get autoremove' to remove them. > > 2014-07-11 02:35:42.796 | The following extra packages will be installed: > > 2014-07-11 02:35:42.796 | cpu-checker ipxe-qemu libbluetooth3 > libbrlapi0.6 > > libcaca0 libfdt1 > > 2014-07-11 02:35:42.796 | libsdl1.2debian libseccomp2 libspice-server1 > > libusbredirparser1 libxen-4.4 > > 2014-07-11 02:35:42.796 | libxenstore3.0 libyajl2 msr-tools > qemu-keymaps > > qemu-system-common > > 2014-07-11 02:35:42.796 | qemu-system-x86 seabios > > 2014-07-11 02:35:42.797 | Suggested packages: > > 2014-07-11 02:35:42.797 | samba vde2 sgabios > > 2014-07-11 02:35:42.798 | The following NEW packages will be installed: > > 2014-07-11 02:35:42.798 | cpu-checker ipxe-qemu libbluetooth3 > libbrlapi0.6 > > libcaca0 libfdt1 > > 2014-07-11 02:35:42.798 | libsdl1.2debian libseccomp2 libspice-server1 > > libusbredirparser1 libxen-4.4 > > 2014-07-11 02:35:42.798 | libxenstore3.0 libyajl2 msr-tools > qemu-keymaps > > qemu-kvm qemu-system-common > > 2014-07-11 02:35:42.798 | qemu-system-x86 seabios > > 2014-07-11 02:35:42.851 | 0 upgraded, 19 newly installed, 0 to remove > and 0 > > not upgraded. > > 2014-07-11 02:35:42.851 | Need to get 291 kB/3985 kB of archives. > > 2014-07-11 02:35:42.851 | After this operation, 20.4 MB of additional > disk > > space will be used. > > 2014-07-11 02:35:42.851 | Err http://mirror.rackspace.com/ubuntu/ > > trusty-security/main libxenstore3.0 amd64 4.4.0-0ubuntu5.1 > > 2014-07-11 02:35:42.851 | 404 Not Found > > 2014-07-11 02:35:42.855 | Err http://mirror.rackspace.com/ubuntu/ > > trusty-security/main libxen-4.4 amd64 4.4.0-0ubuntu5.1 > > 2014-07-11 02:35:42.855 | 404 Not Found > > 2014-07-11 02:35:42.858 | E: Failed to fetch > > > http://mirror.rackspace.com/ubuntu/pool/main/x/xen/libxenstore3.0_4.4.0-0ubuntu5.1_amd64.deb > > 404 Not Found > > 2014-07-11 02:35:42.858 | > > 2014-07-11 02:35:42.858 | E: Failed to fetch > > > http://mirror.rackspace.com/ubuntu/pool/main/x/xen/libxen-4.4_4.4.0-0ubuntu5.1_amd64.deb > > 404 Not Found > > 2014-07-11 02:35:42.858 | > > 2014-07-11 02:35:42.858 | E: Unable to fetch some archives, maybe run > > apt-get update or try with --fix-missing? > > > > -- > > Best Regards > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > This bug is a duplicate of https://bugs.launchpad.net/bugs/1286818 I > have gone ahead and marked it that way. The issue here is the > rackspace mirrors periodically go sideways and don't work properly. We > think this is because they are not syncing from ubuntu safely. > > There are a couple options to fix this. We can run our own ubuntu, > centos, and fedora mirrors. There has been some work to get this > going, https://review.openstack.org/#/c/89928/1 and > https://review.openstack.org/#/c/90875/. pleia2 and dprince should > know more. Rackspace could also correct their mirror syncing. Or we > could possibly point a different mirrors entirely but that probably > won't be any better than rackspace in the long run due to the Internet > being unreliable. The first option gives us the most control and > ability to react if things break. > > Feel free to review and/or update those changes as appropriate. > > Clark > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best Regards, Gengyuan Zhang NetEase Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Infra] Jenkins gate jobs fails
Several jenkins gate jobs failed since some lib packages in ubuntu source are missing and devstack failed to setup the tempest env. Could there be someone help to fix this? I have filed the bug here: https://bugs.launchpad.net/openstack-ci/+bug/1340514 2014-07-11 02:35:42.333 | + apt_get install qemu-kvm 2014-07-11 02:35:42.336 | + sudo DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy= apt-get --option Dpkg::Options::=--force-confold --assume-yes install qemu-kvm 2014-07-11 02:35:42.358 | Reading package lists... 2014-07-11 02:35:42.602 | Building dependency tree... 2014-07-11 02:35:42.604 | Reading state information... 2014-07-11 02:35:42.769 | The following packages were automatically installed and are no longer required: 2014-07-11 02:35:42.769 | python-colorama python-distlib python-html5lib 2014-07-11 02:35:42.769 | Use 'apt-get autoremove' to remove them. 2014-07-11 02:35:42.796 | The following extra packages will be installed: 2014-07-11 02:35:42.796 | cpu-checker ipxe-qemu libbluetooth3 libbrlapi0.6 libcaca0 libfdt1 2014-07-11 02:35:42.796 | libsdl1.2debian libseccomp2 libspice-server1 libusbredirparser1 libxen-4.4 2014-07-11 02:35:42.796 | libxenstore3.0 libyajl2 msr-tools qemu-keymaps qemu-system-common 2014-07-11 02:35:42.796 | qemu-system-x86 seabios 2014-07-11 02:35:42.797 | Suggested packages: 2014-07-11 02:35:42.797 | samba vde2 sgabios 2014-07-11 02:35:42.798 | The following NEW packages will be installed: 2014-07-11 02:35:42.798 | cpu-checker ipxe-qemu libbluetooth3 libbrlapi0.6 libcaca0 libfdt1 2014-07-11 02:35:42.798 | libsdl1.2debian libseccomp2 libspice-server1 libusbredirparser1 libxen-4.4 2014-07-11 02:35:42.798 | libxenstore3.0 libyajl2 msr-tools qemu-keymaps qemu-kvm qemu-system-common 2014-07-11 02:35:42.798 | qemu-system-x86 seabios 2014-07-11 02:35:42.851 | 0 upgraded, 19 newly installed, 0 to remove and 0 not upgraded. 2014-07-11 02:35:42.851 | Need to get 291 kB/3985 kB of archives. 2014-07-11 02:35:42.851 | After this operation, 20.4 MB of additional disk space will be used. 2014-07-11 02:35:42.851 | Err http://mirror.rackspace.com/ubuntu/ trusty-security/main libxenstore3.0 amd64 4.4.0-0ubuntu5.1 2014-07-11 02:35:42.851 | 404 Not Found 2014-07-11 02:35:42.855 | Err http://mirror.rackspace.com/ubuntu/ trusty-security/main libxen-4.4 amd64 4.4.0-0ubuntu5.1 2014-07-11 02:35:42.855 | 404 Not Found 2014-07-11 02:35:42.858 | E: Failed to fetch http://mirror.rackspace.com/ubuntu/pool/main/x/xen/libxenstore3.0_4.4.0-0ubuntu5.1_amd64.deb 404 Not Found 2014-07-11 02:35:42.858 | 2014-07-11 02:35:42.858 | E: Failed to fetch http://mirror.rackspace.com/ubuntu/pool/main/x/xen/libxen-4.4_4.4.0-0ubuntu5.1_amd64.deb 404 Not Found 2014-07-11 02:35:42.858 | 2014-07-11 02:35:42.858 | E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? -- Best Regards ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] is tenant-id in network API?
This patch should fix the problem, sorry for the mistakes above.
On Mon, Jul 7, 2014 at 1:46 PM, Andreas Jaeger wrote:
> On 07/07/2014 05:56 AM, Anne Gentle wrote:
> > Please check to see if this patch fixes the issue:
> >
> > https://review.openstack.org/1050
> >
> >
>
> It should be:
>
> https://review.openstack.org/105060
>
> Andreas
> --
> Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
>GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg)
> GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
--
Best Regards,
Gengyuan Zhang
NetEase Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] is tenant-id in network API?
hi, I use the newest api-ref master branch to generate the neutron API docs
and
there is no tenant_id as part of the url.
I think it is because the api-ref website hasn't updated with the newest
api-ref repo.
On Mon, Jul 7, 2014 at 10:53 AM, Anne Gentle wrote:
> We have an API doc bug to track this issue:
>
> https://bugs.launchpad.net/openstack-api-site/+bug/1334837
>
> Thanks for reporting, have marked it High.
> Anne
>
>
> On Sun, Jul 6, 2014 at 9:41 PM, Damon Wang wrote:
>
>> Agree,
>> http://docs.openstack.org/api/openstack-network/2.0/content/networks.html
>> and http://developer.openstack.org/api-ref-networking-v2.html both have
>> this problem.
>>
>>
>> 2014-07-07 10:25 GMT+08:00 Yongsheng Gong :
>>
>>> Hi,
>>> Today, I found the {tenant_id} is written in neutron API URLs at
>>> http://developer.openstack.org/api-ref-networking-v2.html.
>>>
>>> But when I tried to access it accordingly, it failed.
>>>
>>> I don't think we have tenant_id in network API URL.
>>>
>>>
>>> any Idea?
>>>
>>> regards,
>>> yong sheng gong
>>> UnitedStack Inc.
>>>
>>>
>>> ___
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to : openst...@lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
--
Best Regards,
Gengyuan Zhang
NetEase Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron]One security issue about floating ip
I have filed this bug on nova https://bugs.launchpad.net/nova/+bug/1334938 On Fri, Jun 27, 2014 at 10:19 AM, Yongsheng Gong wrote: > I have reported it on neutron project > https://bugs.launchpad.net/neutron/+bug/1334926 > > > On Fri, Jun 27, 2014 at 5:07 AM, Vishvananda Ishaya > wrote: > >> I missed that going in, but it appears that clean_conntrack is not done on >> disassociate, just during migration. It sounds like we should remove the >> explicit call in migrate, and just always call it from remove_floating_ip. >> >> Vish >> >> On Jun 26, 2014, at 1:48 PM, Brian Haley wrote: >> >> > Signed PGP part >> > I believe nova-network does this by using 'conntrack -D -r $fixed_ip' >> when the >> > floating IP goes away (search for clean_conntrack), Neutron doesn't >> when it >> > removes the floating IP. Seems like it's possible to close most of >> that gap >> > in the l3-agent - when it removes the IP from it's qg- interface it can >> do a >> > similar operation. >> > >> > -Brian >> > >> > On 06/26/2014 03:36 PM, Vishvananda Ishaya wrote: >> > > I believe this will affect nova-network as well. We probably should >> use >> > > something like the linux cutter utility to kill any ongoing >> connections >> > > after we remove the nat rule. >> > > >> > > Vish >> > > >> > > On Jun 25, 2014, at 8:18 PM, Xurong Yang wrote: >> > > >> > >> Hi folks, >> > >> >> > >> After we create an SSH connection to a VM via its floating ip, even >> > >> though we have removed the floating ip association, we can still >> access >> > >> the VM via that connection. Namely, SSH is not disconnected when the >> > >> floating ip is not valid. Any good solution about this security >> issue? >> > >> >> > >> Thanks Xurong Yang ___ >> > >> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > >> > > >> > > >> > > ___ OpenStack-dev mailing >> list >> > > OpenStack-dev@lists.openstack.org >> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > >> > >> > >> > ___ >> > OpenStack-dev mailing list >> > OpenStack-dev@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Best Regards, Gengyuan Zhang NetEase Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
