Re: [openstack-dev] [cinder] [nova] whether the ServiceGroup in Cinder is necessary
On 12/28/2015 05:03 AM, hao wang wrote: > hi, Janice > > This idea seems to me that is useful to detect the state of > cinder-volume process more quickly, but I feel there is another issue > that if the back-end device go to fail you still > can't keep cloud in ha or create volume successfully since the service > is up but device is down. > > So, what I want to say is we maybe need to consider to detect and > report the device state priority[1] and then consider to improve > service if we need that. > > [1]https://review.openstack.org/#/c/252921/ We're already doing something similar in terms of driver initialization state [1]. c-vols with uninitialized drivers will show up as "down". Your idea also seems to make sense to me. https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L474-L481 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] [nova] whether the ServiceGroup in Cinder is necessary
Hi hao wang, Firstly, I agree with you that healthy backend is a requirement for creating volumes, as the same as service-up. So, monitor the state of backend is useful. While, ServiceGroup is only used for detecting state of service quickly, So whether backend is up or not is not taken into consideration for ServiceGroup. So, I think there is no priority between them. Thank you. Janice > hi, Janice > > This idea seems to me that is useful to detect the state of > cinder-volume process more quickly, but I feel there is another issue > that if the back-end device go to fail you still > can't keep cloud in ha or create volume successfully since the service > is up but device is down. > > So, what I want to say is we maybe need to consider to detect and > report the device state priority[1] and then consider to improve > service if we need that. > [1]https://review.openstack.org/#/c/252921/ ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s). If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited. If you have received this mail in error, please delete it and notify us immediately. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] [nova] whether the ServiceGroup in Cinder is necessary
hi, Janice This idea seems to me that is useful to detect the state of cinder-volume process more quickly, but I feel there is another issue that if the back-end device go to fail you still can't keep cloud in ha or create volume successfully since the service is up but device is down. So, what I want to say is we maybe need to consider to detect and report the device state priority[1] and then consider to improve service if we need that. [1]https://review.openstack.org/#/c/252921/ 2015-12-28 9:18 GMT+08:00 : >> Hmm, I see. There's this spec[1] that was discussed in the past with a >> similar proposal. There's a SPEC with some other points on the discussion, I >> think Janice >> forgot to mention. > >> Erlon > >> [1] https://review.openstack.org/#/c/176233/ >> [2] https://review.openstack.org/#/c/258968/ > >> On Tue, Dec 22, 2015 at 12:16 PM, Michał Dulko >> wrote: >> On 12/22/2015 01:29 PM, Erlon Cruz wrote: >> > Hi Li, >> > >> > Can you give a quick background on servicegroups (or links to. The >> > spec you linked only describe the process on Nova to change from what >> > they are using to tooz)? Also, what are the use cases and benefits of >> > using this? >> > >> > Erlon >> > > >> This is simply and idea to be able to use something more sophisticated >> than DB heartbeats to monitor services states. With Tooz implemented for >> that we would be able to use for example ZooKeeper to know about service >> failure in a matter of seconds instead of around a minute. This would >> shrink the window in which c-sch doesn't-know-yet that c-vol failed and >> sends RPC messages to a service that will never answer. I think there >> are more use cases related to service monitoring and failover. > >> Service groups isn't probably a correct name for proposed enhancement - >> we have this concept somehow implemented, but proposed idea seems to be >> related to making it pluggable. > > > Hi Erlon and Michal, > Sorry for response you so late. > > The Cinder ServiceGroup is used for getting the state of services > quickly. > use case such as: >1) As an admin, I want to know each cinder service state, so that > I can take some >actions to keep cloud in high availability if any service is > down. >2) As an user, I want my volumes not to be scheduled to failed > cinder-volume >instances. > My colleague and I, have posted the specs[1] of ServiceGroup in > Cinder. > > Janice > > [1] https://review.openstack.org/#/c/258968/ > > > > > 发件人: Erlon Cruz > 收件人: "OpenStack Development Mailing List (not for usage questions)" > , > 日期: 2015/12/23 04:04 > 主题:Re: [openstack-dev] [cinder] [nova] whether the ServiceGroup in > Cinder is necessary > > > > > Hmm, I see. There's this spec[1] that was discussed in the past with a > similar proposal. There's a SPEC with some other points on the discussion, I > think Janice forgot to mention. > > Erlon > > [1] https://review.openstack.org/#/c/176233/ > [2] https://review.openstack.org/#/c/258968/ > > On Tue, Dec 22, 2015 at 12:16 PM, Michał Dulko > wrote: > On 12/22/2015 01:29 PM, Erlon Cruz wrote: >> Hi Li, >> >> Can you give a quick background on servicegroups (or links to. The >> spec you linked only describe the process on Nova to change from what >> they are using to tooz)? Also, what are the use cases and benefits of >> using this? >> >> Erlon >> > > This is simply and idea to be able to use something more sophisticated > than DB heartbeats to monitor services states. With Tooz implemented for > that we would be able to use for example ZooKeeper to know about service > failure in a matter of seconds instead of around a minute. This would > shrink the window in which c-sch doesn't-know-yet that c-vol failed and > sends RPC messages to a service that will never answer. I think there > are more use cases related to service monitoring and failover. > > Service groups isn't probably a correct name for proposed enhancement - > we have this concept somehow implemented, but proposed idea seems to be > related to making it pluggable. > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___
Re: [openstack-dev] [cinder] [nova] whether the ServiceGroup in Cinder is necessary
> Hmm, I see. There's this spec[1] that was discussed in the past with a similar proposal. There's a SPEC with some other points on the discussion, I think Janice > forgot to mention. > Erlon > [1] https://review.openstack.org/#/c/176233/ > [2] https://review.openstack.org/#/c/258968/ > On Tue, Dec 22, 2015 at 12:16 PM, Michał Dulko wrote: > On 12/22/2015 01:29 PM, Erlon Cruz wrote: > > Hi Li, > > > > Can you give a quick background on servicegroups (or links to. The > > spec you linked only describe the process on Nova to change from what > > they are using to tooz)? Also, what are the use cases and benefits of > > using this? > > > > Erlon > > > This is simply and idea to be able to use something more sophisticated > than DB heartbeats to monitor services states. With Tooz implemented for > that we would be able to use for example ZooKeeper to know about service > failure in a matter of seconds instead of around a minute. This would > shrink the window in which c-sch doesn't-know-yet that c-vol failed and > sends RPC messages to a service that will never answer. I think there > are more use cases related to service monitoring and failover. > Service groups isn't probably a correct name for proposed enhancement - > we have this concept somehow implemented, but proposed idea seems to be > related to making it pluggable. Hi Erlon and Michal, Sorry for response you so late. The Cinder ServiceGroup is used for getting the state of services quickly. use case such as: 1) As an admin, I want to know each cinder service state, so that I can take some actions to keep cloud in high availability if any service is down. 2) As an user, I want my volumes not to be scheduled to failed cinder-volume instances. My colleague and I, have posted the specs[1] of ServiceGroup in Cinder. Janice [1] https://review.openstack.org/#/c/258968/ 发件人: Erlon Cruz 收件人: "OpenStack Development Mailing List (not for usage questions)" , 日期: 2015/12/23 04:04 主题: Re: [openstack-dev] [cinder] [nova] whether the ServiceGroup in Cinder is necessary Hmm, I see. There's this spec[1] that was discussed in the past with a similar proposal. There's a SPEC with some other points on the discussion, I think Janice forgot to mention. Erlon [1] https://review.openstack.org/#/c/176233/ [2] https://review.openstack.org/#/c/258968/ On Tue, Dec 22, 2015 at 12:16 PM, Michał Dulko wrote: On 12/22/2015 01:29 PM, Erlon Cruz wrote: > Hi Li, > > Can you give a quick background on servicegroups (or links to. The > spec you linked only describe the process on Nova to change from what > they are using to tooz)? Also, what are the use cases and benefits of > using this? > > Erlon > This is simply and idea to be able to use something more sophisticated than DB heartbeats to monitor services states. With Tooz implemented for that we would be able to use for example ZooKeeper to know about service failure in a matter of seconds instead of around a minute. This would shrink the window in which c-sch doesn't-know-yet that c-vol failed and sends RPC messages to a service that will never answer. I think there are more use cases related to service monitoring and failover. Service groups isn't probably a correct name for proposed enhancement - we have this concept somehow implemented, but proposed idea seems to be related to making it pluggable. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s). If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited. If you have received this mail in error, please delete it and notify us immediately. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] [nova] whether the ServiceGroup in Cinder is necessary
On 12/22/2015 09:01 PM, Erlon Cruz wrote: > Hmm, I see. There's this spec[1] that was discussed in the past with a > similar proposal. There's a SPEC with some other points on the > discussion, I think Janice forgot to mention. > > Erlon > > [1] https://review.openstack.org/#/c/176233/ > [2] https://review.openstack.org/#/c/258968/ It seems to me that these two are actually not that much related. [1] proposes to use RPC calls to determine if a service is alive, but only in cinder-manage commands. In [2] whole health check mechanism that is used by scheduler is proposed to be replaced by Tooz, which isn't using RPC. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] [nova] whether the ServiceGroup in Cinder is necessary
Hmm, I see. There's this spec[1] that was discussed in the past with a similar proposal. There's a SPEC with some other points on the discussion, I think Janice forgot to mention. Erlon [1] https://review.openstack.org/#/c/176233/ [2] https://review.openstack.org/#/c/258968/ On Tue, Dec 22, 2015 at 12:16 PM, Michał Dulko wrote: > On 12/22/2015 01:29 PM, Erlon Cruz wrote: > > Hi Li, > > > > Can you give a quick background on servicegroups (or links to. The > > spec you linked only describe the process on Nova to change from what > > they are using to tooz)? Also, what are the use cases and benefits of > > using this? > > > > Erlon > > > > This is simply and idea to be able to use something more sophisticated > than DB heartbeats to monitor services states. With Tooz implemented for > that we would be able to use for example ZooKeeper to know about service > failure in a matter of seconds instead of around a minute. This would > shrink the window in which c-sch doesn't-know-yet that c-vol failed and > sends RPC messages to a service that will never answer. I think there > are more use cases related to service monitoring and failover. > > Service groups isn't probably a correct name for proposed enhancement - > we have this concept somehow implemented, but proposed idea seems to be > related to making it pluggable. > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] [nova] whether the ServiceGroup in Cinder is necessary
On 12/22/2015 01:29 PM, Erlon Cruz wrote: > Hi Li, > > Can you give a quick background on servicegroups (or links to. The > spec you linked only describe the process on Nova to change from what > they are using to tooz)? Also, what are the use cases and benefits of > using this? > > Erlon > This is simply and idea to be able to use something more sophisticated than DB heartbeats to monitor services states. With Tooz implemented for that we would be able to use for example ZooKeeper to know about service failure in a matter of seconds instead of around a minute. This would shrink the window in which c-sch doesn't-know-yet that c-vol failed and sends RPC messages to a service that will never answer. I think there are more use cases related to service monitoring and failover. Service groups isn't probably a correct name for proposed enhancement - we have this concept somehow implemented, but proposed idea seems to be related to making it pluggable. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] [nova] whether the ServiceGroup in Cinder is necessary
Hi Li, Can you give a quick background on servicegroups (or links to. The spec you linked only describe the process on Nova to change from what they are using to tooz)? Also, what are the use cases and benefits of using this? Erlon On Tue, Dec 22, 2015 at 8:49 AM, Michał Dulko wrote: > On 12/17/2015 04:49 AM, li.yuanz...@zte.com.cn wrote: > > Hi all, > > > > I'd like to start discussion on whether the servicegroup in Cinder > > is necessary. > > > > Recently, cinder can only support db driver, and doesn't have > > servicegroup concept. > > our team wants to implement the servicegroup feature using on > > tooz[1] library. Like nova[2], when the state of service is required, > > it can be got through servicegroup. > > > > Besides, due to the cinder-volume-active-active-support[3] merged, > > we think it makes the Service Group do more. > > > > Before the cinder-volume-active-active-support was proposed, Cinder > > has no concept of cluster. Therefore, we have a doubt that, if without > > cinder-volume-active-active-support, is it necessary to add feature of > > servicegroup? > > > > Any comments or suggestions? > > > > [1] > > > https://github.com/openstack/tooz/blob/master/doc/source/tutorial/group_membership.rst > > > > [2] > > > https://github.com/openstack/nova-specs/blob/master/specs/liberty/approved/service-group-using-tooz.rst > > > > [3] > > > https://github.com/openstack/cinder-specs/blob/master/specs/mitaka/cinder-volume-active-active-support.rst > > > > > > > > Best Regards, > > Janice > > Hi, > > It will not be possible to use A/A HA configuration of cinder-volume > service with LVM driver. According to latest User Survey [1] this driver > is running in 22% of deployments. ZooKeeper service groups will be still > useful there, as it will allow scheduler to know about failed > services/nodes much quicker and prevent from scheduling volumes there. > > As we have initial Tooz integration [2] already merged for locking > purposes, I think that if we'll be able to implement SG in a > non-intrusive manner (without changing the default behavior) it would be > an interesting option for some deployments. > > [1] http://www.openstack.org/assets/survey/Public-User-Survey-Report.pdf > [2] https://review.openstack.org/#/c/183537/ > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] [nova] whether the ServiceGroup in Cinder is necessary
On 12/17/2015 04:49 AM, li.yuanz...@zte.com.cn wrote: > Hi all, > > I'd like to start discussion on whether the servicegroup in Cinder > is necessary. > > Recently, cinder can only support db driver, and doesn't have > servicegroup concept. > our team wants to implement the servicegroup feature using on > tooz[1] library. Like nova[2], when the state of service is required, > it can be got through servicegroup. > > Besides, due to the cinder-volume-active-active-support[3] merged, > we think it makes the Service Group do more. > > Before the cinder-volume-active-active-support was proposed, Cinder > has no concept of cluster. Therefore, we have a doubt that, if without > cinder-volume-active-active-support, is it necessary to add feature of > servicegroup? > > Any comments or suggestions? > > [1] > https://github.com/openstack/tooz/blob/master/doc/source/tutorial/group_membership.rst > > [2] > https://github.com/openstack/nova-specs/blob/master/specs/liberty/approved/service-group-using-tooz.rst > > [3] > https://github.com/openstack/cinder-specs/blob/master/specs/mitaka/cinder-volume-active-active-support.rst > > > > Best Regards, > Janice Hi, It will not be possible to use A/A HA configuration of cinder-volume service with LVM driver. According to latest User Survey [1] this driver is running in 22% of deployments. ZooKeeper service groups will be still useful there, as it will allow scheduler to know about failed services/nodes much quicker and prevent from scheduling volumes there. As we have initial Tooz integration [2] already merged for locking purposes, I think that if we'll be able to implement SG in a non-intrusive manner (without changing the default behavior) it would be an interesting option for some deployments. [1] http://www.openstack.org/assets/survey/Public-User-Survey-Report.pdf [2] https://review.openstack.org/#/c/183537/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [cinder] [nova] whether the ServiceGroup in Cinder is necessary
Hi all, I'd like to start discussion on whether the servicegroup in Cinder is necessary. Recently, cinder can only support db driver, and doesn't have servicegroup concept. our team wants to implement the servicegroup feature using on tooz[1] library. Like nova[2], when the state of service is required, it can be got through servicegroup. Besides, due to the cinder-volume-active-active-support[3] merged, we think it makes the Service Group do more. Before the cinder-volume-active-active-support was proposed, Cinder has no concept of cluster. Therefore, we have a doubt that, if without cinder-volume-active-active-support, is it necessary to add feature of servicegroup? Any comments or suggestions? [1] https://github.com/openstack/tooz/blob/master/doc/source/tutorial/group_membership.rst [2] https://github.com/openstack/nova-specs/blob/master/specs/liberty/approved/service-group-using-tooz.rst [3] https://github.com/openstack/cinder-specs/blob/master/specs/mitaka/cinder-volume-active-active-support.rst Best Regards, Janice 李媛祯 Li Yuanzhen 无线产品经营部 / 控制器四部Product R&D-Wireless / Controller IV 上海市张江高科技园碧波路889号D楼 D306, 889# Bibo Rd;Pudong District Shanghai,China,201203 E: li.yuanz...@zte.com.cn www.zte.com.cn ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s). If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited. If you have received this mail in error, please delete it and notify us immediately. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
