Re: [openstack-dev] [openstack-community] Give cinder-backup more CPU resources

2018-07-16 Thread Gorka Eguileor 
On 06/07, Amy Marrich wrote:
> Hey,
>
> Forwarding to the Dev list as you may get a better response from there.
>
> Thanks,
>
> Amy (spotz)
>

Hi,

I included a good number of improvements regarding bottlenecks on the
backup service in recent releases, some of them are automatic, and
others need configuration tweaks.  But even in older releases there are
a couple of things that can be done to mitigate/fix this situation,
depending on the release you are using.

In general I recommend:

- Using more than one process:

  Use the "backup_processes" option, if available, to run multiple
  subprocesses (like Alan suggested).

  If this option is not available run multiple backup services on the
  same machine (like Duncan suggests), but use a configuration overlay
  to change the host configuration option. That way you won't have
  problems with one service messing up the others' resources at start
  time.

- Increase the native thread pool size: This is even more critical if
  you are using RBD backend:

  Using the "backend_native_threads_pool_size" configuration option [1],
  if available, we can increase the default size.

  If your deployment doesn't have this option, you can still use
  environmental variable "EVENTLET_THREADPOOL_SIZE" to set it in any
  release.

Cheers,
Gorka.

[1]: 
https://github.com/openstack/cinder/commit/e570436d1cca5cfa89388aec8b2daa63d01d0250

> On Thu, Jul 5, 2018 at 11:30 PM, Keynes Lee/WHQ/Wistron <
> keynes_...@wistron.com> wrote:
>
> > Hi
> >
> >
> >
> > When making “cinder backup-create”
> >
> > We found the process “cinder-backup” use 100% util of 1 CPU core on an
> > OpenStack Controller node.
> >
> > It not just causes a bad backup performance, also make the
> > openstack-cinder-backup unstable.
> >
> > Especially when we make several backup at the same time.
> >
> >
> >
> > The Controller Node has 40 CPU cores.
> >
> > Can we assign more CPU resources to cinder-backup ?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > [image: cid:image007.jpg@01D1747D.DB260110]
> >
> > *Keynes  Lee**李* *俊* *賢*
> >
> > Direct:
> >
> > +886-2-6612-1025
> >
> > Mobile:
> >
> > +886-9-1882-3787
> >
> > Fax:
> >
> > +886-2-6612-1991
> >
> >
> >
> > E-Mail:
> >
> > keynes_...@wistron.com
> >
> >
> >
> >
> >
> >
> > *---*
> >
> > *This email contains confidential or legally privileged information and is
> > for the sole use of its intended recipient. *
> >
> > *Any unauthorized review, use, copying or distribution of this email or
> > the content of this email is strictly prohibited.*
> >
> > *If you are not the intended recipient, you may reply to the sender and
> > should delete this e-mail immediately.*
> >
> >
> > *---*
> >
> > ___
> > Community mailing list
> > commun...@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/community
> >
> >



> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-community] Give cinder-backup more CPU resources

2018-07-06 Thread Jay S Bryant 



On 7/6/2018 9:33 AM, Alan Bishop wrote:


On Fri, Jul 6, 2018 at 10:18 AM Amy Marrich > wrote:


Hey,

Forwarding to the Dev list as you may get a better response from
there.

Thanks,

Amy (spotz)

On Thu, Jul 5, 2018 at 11:30 PM, Keynes Lee/WHQ/Wistron
mailto:keynes_...@wistron.com>> wrote:

Hi

When making “cinder backup-create”

We found the process “cinder-backup” use 100% util of 1 CPU
core on an OpenStack Controller node.

It not just causes a bad backup performance, also make the
openstack-cinder-backup unstable.

Especially when we make several backup at the same time.

The Controller Node has 40 CPU cores.

Can we assign more CPU resources to cinder-backup ?


This has been addressed in [1], but it may not be in the release 
you're using.


[1] 
https://github.com/openstack/cinder/commit/373b52404151d80e83004a37d543f825846edea1




In addition to the change above we also have 
https://review.openstack.org/#/c/537003/ which should also help with the 
stability issues.  That has been backported as far as Pike.


The change for multiple processes is only in master for the Rocky 
release right now.


Jay


Alan

cid:image007.jpg@01D1747D.DB260110



*Keynes  Lee **李俊賢*

Direct:



+886-2-6612-1025

Mobile:



+886-9-1882-3787

Fax:



+886-2-6612-1991



E-Mail:



keynes_...@wistron.com 


*---*

*This email contains confidential or legally privileged
information and is for the sole use of its intended recipient. *

*Any unauthorized review, use, copying or distribution of this
email or the content of this email is strictly prohibited.*

*If you are not the intended recipient, you may reply to the
sender and should delete this e-mail immediately.*


*---*


___
Community mailing list
commun...@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/community


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-community] Give cinder-backup more CPU resources

2018-07-06 Thread Duncan Thomas 
You can run many c-bak processes on one node, which will get fed round
robin, so you should see fairly linear speedup in the many backups case
until you run out of CPUs.

Parallelising a single backup was something I attempted, but python makes
it extremely difficult so there's no useful implementation I'm aware of.

On Fri, 6 Jul 2018, 3:18 pm Amy Marrich,  wrote:

> Hey,
>
> Forwarding to the Dev list as you may get a better response from there.
>
> Thanks,
>
> Amy (spotz)
>
> On Thu, Jul 5, 2018 at 11:30 PM, Keynes Lee/WHQ/Wistron <
> keynes_...@wistron.com> wrote:
>
>> Hi
>>
>>
>>
>> When making “cinder backup-create”
>>
>> We found the process “cinder-backup” use 100% util of 1 CPU core on an
>> OpenStack Controller node.
>>
>> It not just causes a bad backup performance, also make the
>> openstack-cinder-backup unstable.
>>
>> Especially when we make several backup at the same time.
>>
>>
>>
>> The Controller Node has 40 CPU cores.
>>
>> Can we assign more CPU resources to cinder-backup ?
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> [image: cid:image007.jpg@01D1747D.DB260110]
>>
>> *Keynes  Lee**李* *俊* *賢*
>>
>> Direct:
>>
>> +886-2-6612-1025
>>
>> Mobile:
>>
>> +886-9-1882-3787
>>
>> Fax:
>>
>> +886-2-6612-1991
>>
>>
>>
>> E-Mail:
>>
>> keynes_...@wistron.com
>>
>>
>>
>>
>>
>>
>> *---*
>>
>> *This email contains confidential or legally privileged information and
>> is for the sole use of its intended recipient. *
>>
>> *Any unauthorized review, use, copying or distribution of this email or
>> the content of this email is strictly prohibited.*
>>
>> *If you are not the intended recipient, you may reply to the sender and
>> should delete this e-mail immediately.*
>>
>>
>> *---*
>>
>> ___
>> Community mailing list
>> commun...@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/community
>>
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-community] Give cinder-backup more CPU resources

2018-07-06 Thread Alan Bishop 
On Fri, Jul 6, 2018 at 10:18 AM Amy Marrich  wrote:

> Hey,
>
> Forwarding to the Dev list as you may get a better response from there.
>
> Thanks,
>
> Amy (spotz)
>
> On Thu, Jul 5, 2018 at 11:30 PM, Keynes Lee/WHQ/Wistron <
> keynes_...@wistron.com> wrote:
>
>> Hi
>>
>>
>>
>> When making “cinder backup-create”
>>
>> We found the process “cinder-backup” use 100% util of 1 CPU core on an
>> OpenStack Controller node.
>>
>> It not just causes a bad backup performance, also make the
>> openstack-cinder-backup unstable.
>>
>> Especially when we make several backup at the same time.
>>
>>
>>
>> The Controller Node has 40 CPU cores.
>>
>> Can we assign more CPU resources to cinder-backup ?
>>
>
This has been addressed in [1], but it may not be in the release you're
using.

[1]
https://github.com/openstack/cinder/commit/373b52404151d80e83004a37d543f825846edea1

Alan


> [image: cid:image007.jpg@01D1747D.DB260110]
>>
>> *Keynes  Lee**李* *俊* *賢*
>>
>> Direct:
>>
>> +886-2-6612-1025
>>
>> Mobile:
>>
>> +886-9-1882-3787
>>
>> Fax:
>>
>> +886-2-6612-1991
>>
>>
>>
>> E-Mail:
>>
>> keynes_...@wistron.com
>>
>>
>>
>>
>>
>>
>> *---*
>>
>> *This email contains confidential or legally privileged information and
>> is for the sole use of its intended recipient. *
>>
>> *Any unauthorized review, use, copying or distribution of this email or
>> the content of this email is strictly prohibited.*
>>
>> *If you are not the intended recipient, you may reply to the sender and
>> should delete this e-mail immediately.*
>>
>>
>> *---*
>>
>> ___
>> Community mailing list
>> commun...@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/community
>>
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-community] Give cinder-backup more CPU resources

2018-07-06 Thread Amy Marrich 
Hey,

Forwarding to the Dev list as you may get a better response from there.

Thanks,

Amy (spotz)

On Thu, Jul 5, 2018 at 11:30 PM, Keynes Lee/WHQ/Wistron <
keynes_...@wistron.com> wrote:

> Hi
>
>
>
> When making “cinder backup-create”
>
> We found the process “cinder-backup” use 100% util of 1 CPU core on an
> OpenStack Controller node.
>
> It not just causes a bad backup performance, also make the
> openstack-cinder-backup unstable.
>
> Especially when we make several backup at the same time.
>
>
>
> The Controller Node has 40 CPU cores.
>
> Can we assign more CPU resources to cinder-backup ?
>
>
>
>
>
>
>
>
>
>
>
>
>
> [image: cid:image007.jpg@01D1747D.DB260110]
>
> *Keynes  Lee**李* *俊* *賢*
>
> Direct:
>
> +886-2-6612-1025
>
> Mobile:
>
> +886-9-1882-3787
>
> Fax:
>
> +886-2-6612-1991
>
>
>
> E-Mail:
>
> keynes_...@wistron.com
>
>
>
>
>
>
> *---*
>
> *This email contains confidential or legally privileged information and is
> for the sole use of its intended recipient. *
>
> *Any unauthorized review, use, copying or distribution of this email or
> the content of this email is strictly prohibited.*
>
> *If you are not the intended recipient, you may reply to the sender and
> should delete this e-mail immediately.*
>
>
> *---*
>
> ___
> Community mailing list
> commun...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/community
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev