Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Davanum Srinivas]

Julien,

Right. the idea is to expand where we use etcd in openstack for other
things as well. example the os-lively effort from Jay, another one is
the oslo.config calling etcd3 fetch config entries.

Thanks,
Dims

On Thu, Mar 30, 2017 at 1:10 PM, Julien Danjou  wrote:
> On Thu, Mar 30 2017, Davanum Srinivas wrote:
>
>> Julien,
>>
>> Yes, Jay is adding this driver for "etcd3://" using the
>> https://pypi.python.org/pypi/etcd3 library
>> https://review.openstack.org/#/c/447223/
>>
>> and i'll followup with a "etcd3+http://; driver using the
>> https://pypi.python.org/pypi/etcd3gw library
>
> Ah ok, works for me. I'm not sure I see the value of using etc3gw which
> is IIUC just a wrapper around requests. For the etcd:// driver in tooz
> we just went ahead and used requests directly, so I imagine that's what
> I'd do for etcd3+http:// :-)
>
> --
> Julien Danjou
> -- Free Software hacker
> -- https://julien.danjou.info



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Julien Danjou]

On Thu, Mar 30 2017, Davanum Srinivas wrote:

> Julien,
>
> Yes, Jay is adding this driver for "etcd3://" using the
> https://pypi.python.org/pypi/etcd3 library
> https://review.openstack.org/#/c/447223/
>
> and i'll followup with a "etcd3+http://; driver using the
> https://pypi.python.org/pypi/etcd3gw library

Ah ok, works for me. I'm not sure I see the value of using etc3gw which
is IIUC just a wrapper around requests. For the etcd:// driver in tooz
we just went ahead and used requests directly, so I imagine that's what
I'd do for etcd3+http:// :-)

-- 
Julien Danjou
-- Free Software hacker
-- https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Davanum Srinivas]

Julien,

Yes, Jay is adding this driver for "etcd3://" using the
https://pypi.python.org/pypi/etcd3 library
https://review.openstack.org/#/c/447223/

and i'll followup with a "etcd3+http://; driver using the
https://pypi.python.org/pypi/etcd3gw library

Thanks,
Dims

On Thu, Mar 30, 2017 at 12:51 PM, Julien Danjou  wrote:
> On Thu, Mar 30 2017, Davanum Srinivas wrote:
>
>> Update.. So i just proposed a new library built off the [3] gist to be
>> added to g-r and u-c.
>>
>> https://review.openstack.org/#/c/450967/
>>
>> The alternative was to add support for custom backends in etcd3
>> library itself with grpc and v3alpha HTTP API as implementations.
>> Since tooz is already an abstraction and the work non-trivial we can
>> start this way. If etcd3 ends up with supporting both, then we can
>> just eliminate this library at that time.
>
> Is it simpler than just writing two tooz drivers, e.g. etcd3 and
> etcd3+http for example?
>
> --
> Julien Danjou
> /* Free Software hacker
>https://julien.danjou.info */



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Julien Danjou]

On Thu, Mar 30 2017, Davanum Srinivas wrote:

> Update.. So i just proposed a new library built off the [3] gist to be
> added to g-r and u-c.
>
> https://review.openstack.org/#/c/450967/
>
> The alternative was to add support for custom backends in etcd3
> library itself with grpc and v3alpha HTTP API as implementations.
> Since tooz is already an abstraction and the work non-trivial we can
> start this way. If etcd3 ends up with supporting both, then we can
> just eliminate this library at that time.

Is it simpler than just writing two tooz drivers, e.g. etcd3 and
etcd3+http for example?

-- 
Julien Danjou
/* Free Software hacker
   https://julien.danjou.info */


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Davanum Srinivas]

Update.. So i just proposed a new library built off the [3] gist to be
added to g-r and u-c.

https://review.openstack.org/#/c/450967/

The alternative was to add support for custom backends in etcd3
library itself with grpc and v3alpha HTTP API as implementations.
Since tooz is already an abstraction and the work non-trivial we can
start this way. If etcd3 ends up with supporting both, then we can
just eliminate this library at that time.

Thanks,
Dims


On Sun, Mar 19, 2017 at 4:34 PM, Davanum Srinivas  wrote:
> Quick update: We have 3 options to talk to etcd:
>
> 1) existing V2 HTTP API - still not deprecated in etcd 3.1.x, so
> existing code in tooz still works.
> 2) grpc based V3 API - We can use the etcd3 package, discussion in
> review[1], problem is that this will not work currently with eventlet
> based services
> 3) v3alpha HTTP API - See details in [2], quick prototype requests
> based code [3]
>
> Thanks,
> Dims
>
> [1] https://review.openstack.org/#/c/446983/
> [2] 
> https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/api_grpc_gateway.md
> [3] https://gist.github.com/dims/19ceaf9472ef54aa3011d7a11e809371
>
> On Sun, Mar 19, 2017 at 9:32 AM, Jay Pipes  wrote:
>> On 03/18/2017 07:48 PM, Mike Perez wrote:
>>>
>>> On 12:35 Mar 14, Jay Pipes wrote:

 On 03/14/2017 08:57 AM, Julien Danjou wrote:
>
> On Tue, Mar 14 2017, Davanum Srinivas wrote:
>
>> Let's do it!! (etcd v2-v3 in tooz)
>
>
> Hehe. I'll move that higher in my priority list, I swear. But anyone is
> free to beat me to it in the meantime. ;)


 A weekend experiment:

 https://github.com/jaypipes/os-lively

 Not tooz, because I'm not interested in a DLM nor leader election library
 (that's what the underlying etcd3 cluster handles for me), only a fast
 service liveness/healthcheck system, but it shows usage of etcd3 and
 Google
 Protocol Buffers implementing a simple API for liveness checking and host
 maintenance reporting.

 My plan is to push some proof-of-concept patches that replace Nova's
 servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
 service liveness checking, which should dramatically reduce the amount of
 both DB traffic as well as conductor/MQ service update traffic.
>>>
>>>
>>> As Monty has mentioned, I'd love for us to decide on one thing. As being
>>> a moderator of that discussion I was trying not to be one-sided.
>>>
>>> Whether or not a decision was made 1.5 years ago, the community that was
>>> present at that time of the decision at the summit decided on an
>>> abstraction
>>> layer to have options. Forcing an option on the community without
>>> gathering
>>> feedback of what the community currently looks like is not a good idea.
>>>
>>> I'd recommend if you want to make this base service, start the discussions
>>> in
>>> somewhere other than the dev list, like the Forum.
>>
>>
>> Mike, it was an experiment :)
>>
>> But, yes, happy to participate in a discussion at the forum.
>>
>>
>> Best,
>> -jay
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Davanum Srinivas :: https://twitter.com/dims



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Jay Pipes]

Also, I gave a stab at an etcd3 tooz coordination driver:

https://review.openstack.org/#/c/447223/

Can't figure out how to get the tests to run properly, but meh, I'll get 
to it in a bit :)


-jay

On 03/19/2017 04:34 PM, Davanum Srinivas wrote:

Quick update: We have 3 options to talk to etcd:

1) existing V2 HTTP API - still not deprecated in etcd 3.1.x, so
existing code in tooz still works.
2) grpc based V3 API - We can use the etcd3 package, discussion in
review[1], problem is that this will not work currently with eventlet
based services
3) v3alpha HTTP API - See details in [2], quick prototype requests
based code [3]

Thanks,
Dims

[1] https://review.openstack.org/#/c/446983/
[2] 
https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/api_grpc_gateway.md
[3] https://gist.github.com/dims/19ceaf9472ef54aa3011d7a11e809371

On Sun, Mar 19, 2017 at 9:32 AM, Jay Pipes  wrote:

On 03/18/2017 07:48 PM, Mike Perez wrote:


On 12:35 Mar 14, Jay Pipes wrote:


On 03/14/2017 08:57 AM, Julien Danjou wrote:


On Tue, Mar 14 2017, Davanum Srinivas wrote:


Let's do it!! (etcd v2-v3 in tooz)



Hehe. I'll move that higher in my priority list, I swear. But anyone is
free to beat me to it in the meantime. ;)



A weekend experiment:

https://github.com/jaypipes/os-lively

Not tooz, because I'm not interested in a DLM nor leader election library
(that's what the underlying etcd3 cluster handles for me), only a fast
service liveness/healthcheck system, but it shows usage of etcd3 and
Google
Protocol Buffers implementing a simple API for liveness checking and host
maintenance reporting.

My plan is to push some proof-of-concept patches that replace Nova's
servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
service liveness checking, which should dramatically reduce the amount of
both DB traffic as well as conductor/MQ service update traffic.



As Monty has mentioned, I'd love for us to decide on one thing. As being
a moderator of that discussion I was trying not to be one-sided.

Whether or not a decision was made 1.5 years ago, the community that was
present at that time of the decision at the summit decided on an
abstraction
layer to have options. Forcing an option on the community without
gathering
feedback of what the community currently looks like is not a good idea.

I'd recommend if you want to make this base service, start the discussions
in
somewhere other than the dev list, like the Forum.



Mike, it was an experiment :)

But, yes, happy to participate in a discussion at the forum.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Davanum Srinivas]

Quick update: We have 3 options to talk to etcd:

1) existing V2 HTTP API - still not deprecated in etcd 3.1.x, so
existing code in tooz still works.
2) grpc based V3 API - We can use the etcd3 package, discussion in
review[1], problem is that this will not work currently with eventlet
based services
3) v3alpha HTTP API - See details in [2], quick prototype requests
based code [3]

Thanks,
Dims

[1] https://review.openstack.org/#/c/446983/
[2] 
https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/api_grpc_gateway.md
[3] https://gist.github.com/dims/19ceaf9472ef54aa3011d7a11e809371

On Sun, Mar 19, 2017 at 9:32 AM, Jay Pipes  wrote:
> On 03/18/2017 07:48 PM, Mike Perez wrote:
>>
>> On 12:35 Mar 14, Jay Pipes wrote:
>>>
>>> On 03/14/2017 08:57 AM, Julien Danjou wrote:

 On Tue, Mar 14 2017, Davanum Srinivas wrote:

> Let's do it!! (etcd v2-v3 in tooz)


 Hehe. I'll move that higher in my priority list, I swear. But anyone is
 free to beat me to it in the meantime. ;)
>>>
>>>
>>> A weekend experiment:
>>>
>>> https://github.com/jaypipes/os-lively
>>>
>>> Not tooz, because I'm not interested in a DLM nor leader election library
>>> (that's what the underlying etcd3 cluster handles for me), only a fast
>>> service liveness/healthcheck system, but it shows usage of etcd3 and
>>> Google
>>> Protocol Buffers implementing a simple API for liveness checking and host
>>> maintenance reporting.
>>>
>>> My plan is to push some proof-of-concept patches that replace Nova's
>>> servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
>>> service liveness checking, which should dramatically reduce the amount of
>>> both DB traffic as well as conductor/MQ service update traffic.
>>
>>
>> As Monty has mentioned, I'd love for us to decide on one thing. As being
>> a moderator of that discussion I was trying not to be one-sided.
>>
>> Whether or not a decision was made 1.5 years ago, the community that was
>> present at that time of the decision at the summit decided on an
>> abstraction
>> layer to have options. Forcing an option on the community without
>> gathering
>> feedback of what the community currently looks like is not a good idea.
>>
>> I'd recommend if you want to make this base service, start the discussions
>> in
>> somewhere other than the dev list, like the Forum.
>
>
> Mike, it was an experiment :)
>
> But, yes, happy to participate in a discussion at the forum.
>
>
> Best,
> -jay
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Jay Pipes]

On 03/18/2017 07:48 PM, Mike Perez wrote:

On 12:35 Mar 14, Jay Pipes wrote:

On 03/14/2017 08:57 AM, Julien Danjou wrote:

On Tue, Mar 14 2017, Davanum Srinivas wrote:


Let's do it!! (etcd v2-v3 in tooz)


Hehe. I'll move that higher in my priority list, I swear. But anyone is
free to beat me to it in the meantime. ;)


A weekend experiment:

https://github.com/jaypipes/os-lively

Not tooz, because I'm not interested in a DLM nor leader election library
(that's what the underlying etcd3 cluster handles for me), only a fast
service liveness/healthcheck system, but it shows usage of etcd3 and Google
Protocol Buffers implementing a simple API for liveness checking and host
maintenance reporting.

My plan is to push some proof-of-concept patches that replace Nova's
servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
service liveness checking, which should dramatically reduce the amount of
both DB traffic as well as conductor/MQ service update traffic.


As Monty has mentioned, I'd love for us to decide on one thing. As being
a moderator of that discussion I was trying not to be one-sided.

Whether or not a decision was made 1.5 years ago, the community that was
present at that time of the decision at the summit decided on an abstraction
layer to have options. Forcing an option on the community without gathering
feedback of what the community currently looks like is not a good idea.

I'd recommend if you want to make this base service, start the discussions in
somewhere other than the dev list, like the Forum.


Mike, it was an experiment :)

But, yes, happy to participate in a discussion at the forum.

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Mike Perez]

On 12:35 Mar 14, Jay Pipes wrote:
> On 03/14/2017 08:57 AM, Julien Danjou wrote:
> >On Tue, Mar 14 2017, Davanum Srinivas wrote:
> >
> >>Let's do it!! (etcd v2-v3 in tooz)
> >
> >Hehe. I'll move that higher in my priority list, I swear. But anyone is
> >free to beat me to it in the meantime. ;)
> 
> A weekend experiment:
> 
> https://github.com/jaypipes/os-lively
> 
> Not tooz, because I'm not interested in a DLM nor leader election library
> (that's what the underlying etcd3 cluster handles for me), only a fast
> service liveness/healthcheck system, but it shows usage of etcd3 and Google
> Protocol Buffers implementing a simple API for liveness checking and host
> maintenance reporting.
> 
> My plan is to push some proof-of-concept patches that replace Nova's
> servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
> service liveness checking, which should dramatically reduce the amount of
> both DB traffic as well as conductor/MQ service update traffic.

As Monty has mentioned, I'd love for us to decide on one thing. As being
a moderator of that discussion I was trying not to be one-sided.

Whether or not a decision was made 1.5 years ago, the community that was
present at that time of the decision at the summit decided on an abstraction
layer to have options. Forcing an option on the community without gathering
feedback of what the community currently looks like is not a good idea.

I'd recommend if you want to make this base service, start the discussions in
somewhere other than the dev list, like the Forum.

-- 
Mike Perez


pgpKfEPbE8x16.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Mike Perez]

On 16:37 Mar 18, Mike Perez wrote:
> On 16:02 Mar 15, Monty Taylor wrote:
> > On 03/15/2017 02:52 PM, Louis Taylor wrote:
> > > On Wed, Mar 15, 2017 at 1:44 PM, Julien Danjou  wrote:
> > >> On Wed, Mar 15 2017, Davanum Srinivas wrote:
> > >>
> > >>> Yep, jd__ and i confirmed that things work with 3.x
> > >>
> > >> Though to be clear, what's used in tooz is the v2 HTTP API, not the new
> > >> v3 gRPC API.
> > > 
> > > And just to be double clear: although etcd 3.x comes with a v2 api,
> > > the etcd3 api also has a different data model, so the data stored
> > > using it is not accessible to the v2 api, and vice versa.
> > 
> > AH! That's different...
> > 
> > For my money, not that I have any, I'd still suggest we just pile on
> > getting moved over to v3 gRPC real quick and then make that the default.
> 
> I'm also happy about this move. However, it doesn't matter if we made this
> decision 1.5 years ago. What matters is we this decision in a place that 
> others
> were around to contribute to the decision, which was the summit. The Forum is
> coming up, so I have proposed the topic. Coming to a decision of changing the
> default or even a base service for our operators over the dev list is not
> a good idea.

Thinking about this some more, I guess changing the default in gate won't
matter as long as some implementation of tooz is being used. My mind is fuzzy
whether or not we explicity decided to gate on zookeeper, or if what Monty says
is right that it was the only thing working so we did that. Removed my -1.

-- 
Mike Perez


pgprG61VgrOgM.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Mike Perez]

On 16:02 Mar 15, Monty Taylor wrote:
> On 03/15/2017 02:52 PM, Louis Taylor wrote:
> > On Wed, Mar 15, 2017 at 1:44 PM, Julien Danjou  wrote:
> >> On Wed, Mar 15 2017, Davanum Srinivas wrote:
> >>
> >>> Yep, jd__ and i confirmed that things work with 3.x
> >>
> >> Though to be clear, what's used in tooz is the v2 HTTP API, not the new
> >> v3 gRPC API.
> > 
> > And just to be double clear: although etcd 3.x comes with a v2 api,
> > the etcd3 api also has a different data model, so the data stored
> > using it is not accessible to the v2 api, and vice versa.
> 
> AH! That's different...
> 
> For my money, not that I have any, I'd still suggest we just pile on
> getting moved over to v3 gRPC real quick and then make that the default.

I'm also happy about this move. However, it doesn't matter if we made this
decision 1.5 years ago. What matters is we this decision in a place that others
were around to contribute to the decision, which was the summit. The Forum is
coming up, so I have proposed the topic. Coming to a decision of changing the
default or even a base service for our operators over the dev list is not
a good idea.

-- 
Mike Perez


pgpQN_PCgFfuX.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Clint Byrum]

Excerpts from Clark Boylan's message of 2017-03-15 09:50:49 -0700:
> On Wed, Mar 15, 2017, at 06:19 AM, Monty Taylor wrote:
> > On 03/15/2017 11:37 AM, Davanum Srinivas wrote:
> > > Monty, Team,
> > > 
> > > Sorry for the top post:
> > > 
> > > Support for etcd/tooz in devstack (with file driver as default) -
> > > https://review.openstack.org/#/c/445432/
> > > 
> > > As of right now both zookeeper driver and etcd driver is working fine:
> > > https://review.openstack.org/#/c/445630/
> > > https://review.openstack.org/#/c/445629/
> > > 
> > > The problem we have from before is that we do not have any CI jobs
> > > that used zookeeper.
> > > 
> > > I am leaning towards just throwing the etcd as default and if folks
> > > are interested in zookeeper then they can add specific CI jobs with
> > > DLM_BACKEND variable set.
> > 
> > That doesn't bother me - zk as the default choice was because at the
> > time zk worked and etcd did not.
> > 
> > That said - etcd3 is a newer/better thing - so maybe instead of driving
> > etcd home as a default before we add etcd3 support, we just change tooz
> > to support etcd3, add the devstack jobs to use that, and start from a
> > position that doesn't involve dealing with any legacy?
> > 
> 
> One logistical concern that no one else seems to have pointed out on
> this thread yet is that the example devstack setup linked at the
> beginning of the thread
> (http://git.openstack.org/cgit/openstack/dragonflow/tree/devstack/etcd_driver)
> grabs tarball from github to perform the etcd3 installation. Looks like
> Fedora and CentOS may have proper packages for etcd3 but Ubuntu does
> not.
> 
> For reliability reasons we probably do not want to be grabbing this
> tarball from github on every job run (particularly if this becomes a
> "base service" installed on every devstack job run).
> 

Zesty looks to be getting etcd3, though there are dependencies that
haven't synced/built yet[1].  Once that's done we can submit a backport and
get it available on Xenial and then just 'apt-get install backports/etcd'

Of course, using backports in LTS's is a little bit weird since the
security team won't maintain backports.

[1] https://launchpad.net/ubuntu/+source/etcd/3.1.0-1/+build/12016435

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Clark Boylan]

On Wed, Mar 15, 2017, at 06:19 AM, Monty Taylor wrote:
> On 03/15/2017 11:37 AM, Davanum Srinivas wrote:
> > Monty, Team,
> > 
> > Sorry for the top post:
> > 
> > Support for etcd/tooz in devstack (with file driver as default) -
> > https://review.openstack.org/#/c/445432/
> > 
> > As of right now both zookeeper driver and etcd driver is working fine:
> > https://review.openstack.org/#/c/445630/
> > https://review.openstack.org/#/c/445629/
> > 
> > The problem we have from before is that we do not have any CI jobs
> > that used zookeeper.
> > 
> > I am leaning towards just throwing the etcd as default and if folks
> > are interested in zookeeper then they can add specific CI jobs with
> > DLM_BACKEND variable set.
> 
> That doesn't bother me - zk as the default choice was because at the
> time zk worked and etcd did not.
> 
> That said - etcd3 is a newer/better thing - so maybe instead of driving
> etcd home as a default before we add etcd3 support, we just change tooz
> to support etcd3, add the devstack jobs to use that, and start from a
> position that doesn't involve dealing with any legacy?
> 

One logistical concern that no one else seems to have pointed out on
this thread yet is that the example devstack setup linked at the
beginning of the thread
(http://git.openstack.org/cgit/openstack/dragonflow/tree/devstack/etcd_driver)
grabs tarball from github to perform the etcd3 installation. Looks like
Fedora and CentOS may have proper packages for etcd3 but Ubuntu does
not.

For reliability reasons we probably do not want to be grabbing this
tarball from github on every job run (particularly if this becomes a
"base service" installed on every devstack job run).

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Julien Danjou]

On Wed, Mar 15 2017, Monty Taylor wrote:

> On 03/15/2017 02:44 PM, Julien Danjou wrote:
>> On Wed, Mar 15 2017, Davanum Srinivas wrote:
>> 
>>> Yep, jd__ and i confirmed that things work with 3.x
>> 
>> Though to be clear, what's used in tooz is the v2 HTTP API, not the new
>> v3 gRPC API.
>
> But if it conceptually works with v3 server using v2 http api, then we
> should be able to iterate in support for grpc api as patches to tooz and
> have everything be happy, right?

Definitely!

-- 
Julien Danjou
-- Free Software hacker
-- https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Monty Taylor]

On 03/15/2017 02:52 PM, Louis Taylor wrote:
> On Wed, Mar 15, 2017 at 1:44 PM, Julien Danjou  wrote:
>> On Wed, Mar 15 2017, Davanum Srinivas wrote:
>>
>>> Yep, jd__ and i confirmed that things work with 3.x
>>
>> Though to be clear, what's used in tooz is the v2 HTTP API, not the new
>> v3 gRPC API.
> 
> And just to be double clear: although etcd 3.x comes with a v2 api,
> the etcd3 api also has a different data model, so the data stored
> using it is not accessible to the v2 api, and vice versa.

AH! That's different...

For my money, not that I have any, I'd still suggest we just pile on
getting moved over to v3 gRPC real quick and then make that the default.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Monty Taylor]

On 03/15/2017 02:44 PM, Julien Danjou wrote:
> On Wed, Mar 15 2017, Davanum Srinivas wrote:
> 
>> Yep, jd__ and i confirmed that things work with 3.x
> 
> Though to be clear, what's used in tooz is the v2 HTTP API, not the new
> v3 gRPC API.

But if it conceptually works with v3 server using v2 http api, then we
should be able to iterate in support for grpc api as patches to tooz and
have everything be happy, right?




signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Monty Taylor]

On 03/15/2017 02:26 PM, Davanum Srinivas wrote:
> On Wed, Mar 15, 2017 at 9:19 AM, Monty Taylor  wrote:
>> On 03/15/2017 11:37 AM, Davanum Srinivas wrote:
>>> Monty, Team,
>>>
>>> Sorry for the top post:
>>>
>>> Support for etcd/tooz in devstack (with file driver as default) -
>>> https://review.openstack.org/#/c/445432/
>>>
>>> As of right now both zookeeper driver and etcd driver is working fine:
>>> https://review.openstack.org/#/c/445630/
>>> https://review.openstack.org/#/c/445629/
>>>
>>> The problem we have from before is that we do not have any CI jobs
>>> that used zookeeper.
>>>
>>> I am leaning towards just throwing the etcd as default and if folks
>>> are interested in zookeeper then they can add specific CI jobs with
>>> DLM_BACKEND variable set.
>>
>> That doesn't bother me - zk as the default choice was because at the
>> time zk worked and etcd did not.
>>
>> That said - etcd3 is a newer/better thing - so maybe instead of driving
>> etcd home as a default before we add etcd3 support, we just change tooz
>> to support etcd3, add the devstack jobs to use that, and start from a
>> position that doesn't involve dealing with any legacy?
> 
> Yep, jd__ and i confirmed that things work with 3.x

WOOT!

> Thanks,
> Dims
> 
>>
>>> On Tue, Mar 14, 2017 at 11:00 PM, Monty Taylor  wrote:
 On 03/15/2017 03:13 AM, Jay Pipes wrote:
> On 03/14/2017 05:01 PM, Clint Byrum wrote:
>> Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
>>> On 03/14/2017 02:50 PM, Julien Danjou wrote:
 On Tue, Mar 14 2017, Jay Pipes wrote:

> Not tooz, because I'm not interested in a DLM nor leader election
> library
> (that's what the underlying etcd3 cluster handles for me), only a
> fast service
> liveness/healthcheck system, but it shows usage of etcd3 and Google
> Protocol
> Buffers implementing a simple API for liveness checking and host
> maintenance
> reporting.

 Cool cool. So that's the same feature that we implemented in tooz 3
 years ago. It's called "group membership". You create a group, make
 nodes join it, and you know who's dead/alive and get notified when
 their
 status change.
>>>
>>> The point of os-lively is not to provide a thin API over ZooKeeper's
>>> group membership interface. The point of os-lively is to remove the need
>>> to have a database (RDBMS) record of a service in Nova.
>>
>> That's also the point of tooz's group membership API:
>>
>> https://docs.openstack.org/developer/tooz/compatibility.html#grouping
>
> Did you take a look at the code I wrote in os-lively? What part of the
> tooz group membership API do you think I would have used?
>
> Again, this was a weekend project that I was moving fast on. I looked at
> tooz and didn't see how I could use it for my purposes, which was to
> store a versioned object in a consistent key/value store with support
> for transactional semantics when storing index and data records at the
> same time [1]
>
> https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511
>
>
> etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional
> semantics in a consistent key/value store that I needed.
>
> tooz is cool, but it's not what I was looking for. It's solving a
> different problem than I was trying to solve.
>
> This isn't a case of NIH, despite what Julien is trying to intimate in
> his emails.
>
>>> tooz simply abstracts a group membership API across a number of drivers.
>>> I don't need that. I need a way to maintain a service record (with
>>> maintenance period information, region, and an evolvable data record
>>> format) and query those service records in an RDBMS-like manner but
>>> without the RDBMS being involved.
>>>
> servicegroup API with os-lively and eliminate Nova's use of an
> RDBMS for
> service liveness checking, which should dramatically reduce the
> amount of both
> DB traffic as well as conductor/MQ service update traffic.

 Interesting. Joshua and Vilob tried to push usage of tooz group
 membership a couple of years ago, but it got nowhere. Well, no, they
 got
 2 specs written IIRC:


 https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html


 But then it died for whatever reasons on Nova side.
>>>
>>> It died because it didn't actually solve a problem.
>>>
>>> The problem is that even if we incorporate tooz, we would still need to
>>> have a service table in the RDBMS and continue to query it over and over
>>> again in the scheduler and API nodes.
>>
>> Most 

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Louis Taylor]

On Wed, Mar 15, 2017 at 1:44 PM, Julien Danjou  wrote:
> On Wed, Mar 15 2017, Davanum Srinivas wrote:
>
>> Yep, jd__ and i confirmed that things work with 3.x
>
> Though to be clear, what's used in tooz is the v2 HTTP API, not the new
> v3 gRPC API.

And just to be double clear: although etcd 3.x comes with a v2 api,
the etcd3 api also has a different data model, so the data stored
using it is not accessible to the v2 api, and vice versa.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Julien Danjou]

On Wed, Mar 15 2017, Davanum Srinivas wrote:

> Yep, jd__ and i confirmed that things work with 3.x

Though to be clear, what's used in tooz is the v2 HTTP API, not the new
v3 gRPC API.

-- 
Julien Danjou
# Free Software hacker
# https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Davanum Srinivas]

On Wed, Mar 15, 2017 at 9:19 AM, Monty Taylor  wrote:
> On 03/15/2017 11:37 AM, Davanum Srinivas wrote:
>> Monty, Team,
>>
>> Sorry for the top post:
>>
>> Support for etcd/tooz in devstack (with file driver as default) -
>> https://review.openstack.org/#/c/445432/
>>
>> As of right now both zookeeper driver and etcd driver is working fine:
>> https://review.openstack.org/#/c/445630/
>> https://review.openstack.org/#/c/445629/
>>
>> The problem we have from before is that we do not have any CI jobs
>> that used zookeeper.
>>
>> I am leaning towards just throwing the etcd as default and if folks
>> are interested in zookeeper then they can add specific CI jobs with
>> DLM_BACKEND variable set.
>
> That doesn't bother me - zk as the default choice was because at the
> time zk worked and etcd did not.
>
> That said - etcd3 is a newer/better thing - so maybe instead of driving
> etcd home as a default before we add etcd3 support, we just change tooz
> to support etcd3, add the devstack jobs to use that, and start from a
> position that doesn't involve dealing with any legacy?

Yep, jd__ and i confirmed that things work with 3.x

Thanks,
Dims

>
>> On Tue, Mar 14, 2017 at 11:00 PM, Monty Taylor  wrote:
>>> On 03/15/2017 03:13 AM, Jay Pipes wrote:
 On 03/14/2017 05:01 PM, Clint Byrum wrote:
> Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
>> On 03/14/2017 02:50 PM, Julien Danjou wrote:
>>> On Tue, Mar 14 2017, Jay Pipes wrote:
>>>
 Not tooz, because I'm not interested in a DLM nor leader election
 library
 (that's what the underlying etcd3 cluster handles for me), only a
 fast service
 liveness/healthcheck system, but it shows usage of etcd3 and Google
 Protocol
 Buffers implementing a simple API for liveness checking and host
 maintenance
 reporting.
>>>
>>> Cool cool. So that's the same feature that we implemented in tooz 3
>>> years ago. It's called "group membership". You create a group, make
>>> nodes join it, and you know who's dead/alive and get notified when
>>> their
>>> status change.
>>
>> The point of os-lively is not to provide a thin API over ZooKeeper's
>> group membership interface. The point of os-lively is to remove the need
>> to have a database (RDBMS) record of a service in Nova.
>
> That's also the point of tooz's group membership API:
>
> https://docs.openstack.org/developer/tooz/compatibility.html#grouping

 Did you take a look at the code I wrote in os-lively? What part of the
 tooz group membership API do you think I would have used?

 Again, this was a weekend project that I was moving fast on. I looked at
 tooz and didn't see how I could use it for my purposes, which was to
 store a versioned object in a consistent key/value store with support
 for transactional semantics when storing index and data records at the
 same time [1]

 https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511


 etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional
 semantics in a consistent key/value store that I needed.

 tooz is cool, but it's not what I was looking for. It's solving a
 different problem than I was trying to solve.

 This isn't a case of NIH, despite what Julien is trying to intimate in
 his emails.

>> tooz simply abstracts a group membership API across a number of drivers.
>> I don't need that. I need a way to maintain a service record (with
>> maintenance period information, region, and an evolvable data record
>> format) and query those service records in an RDBMS-like manner but
>> without the RDBMS being involved.
>>
 servicegroup API with os-lively and eliminate Nova's use of an
 RDBMS for
 service liveness checking, which should dramatically reduce the
 amount of both
 DB traffic as well as conductor/MQ service update traffic.
>>>
>>> Interesting. Joshua and Vilob tried to push usage of tooz group
>>> membership a couple of years ago, but it got nowhere. Well, no, they
>>> got
>>> 2 specs written IIRC:
>>>
>>>
>>> https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
>>>
>>>
>>> But then it died for whatever reasons on Nova side.
>>
>> It died because it didn't actually solve a problem.
>>
>> The problem is that even if we incorporate tooz, we would still need to
>> have a service table in the RDBMS and continue to query it over and over
>> again in the scheduler and API nodes.
>
> Most likely it was designed with hesitance to have a tooz requirement
> to be a source of truth. But it's certainly not a problem for most tooz
> backends to be a source of 

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Monty Taylor]

On 03/15/2017 01:48 PM, John Garbutt wrote:
> On 15 March 2017 at 12:33, Sean Dague  wrote:
>> On 03/15/2017 08:10 AM, John Garbutt wrote:
>>> On 15 March 2017 at 11:58, Jay Pipes  wrote:
 On 03/15/2017 07:44 AM, Sean Dague wrote:
>
> On 03/14/2017 11:00 PM, Monty Taylor wrote:
> 
>>
>> a) awesome. when the rest of this dips momentarily into words that might
>> sound negative, please hear it all wrapped in an "awesome" and know that
>> my personal desire is to see the thing you're working on be successful
>> without undue burden...
>>
>> b) In Tokyo, we had the big discussion about DLMs (where at least my
>> intent going in to the room was to get us to pick one and only one).
>> There were three camps in the room who were all vocal:
>>
>> 1) YES! Let's just pick one, I don't care which one
>> 2) I hate Java I don't want to run Zookeeper, so we can't pick that
>> 3) I hate go/don't trust coreos I don't want to run etcd so we can't
>> pick that
>>
>> Because of 2 and 3 the group represented by 1 lost and we ended up with:
>> "crap, we have to use an abstraction library"
>>
>> I'd argue that unless something has changed significantly, having Nova
>> grow a direct depend on etcd when the DLM discussion brought us to "the
>> operators in the room have expressed a need for a pluggable choice
>> between at least zk and etcd" should be pretty much a non-starter.
>>
>> Now, being that I was personally in group 1, I'd be THRILLED if we
>> could, as a community, decide to pick one and skip having an abstraction
>> library. I still don't care which one - and you know I love
>> gRPC/protobuf.
>>
>> But I do think that given the anti-etcd sentiment that was expressed was
>> equally as vehement as the anti-zk sentiment, that we need to circle
>> back and make a legit call on this topic.
>>
>> If we can pick one, I think having special-purpose libraries like
>> os-lively for specific purposes would be neat.
>>
>> If we still can't pick one, then I think adding the liveness check you
>> implemented for os-lively as a new feature in tooz and also implementing
>> the same thing in the zk driver would be necessary. (of course, that'll
>> probably depend on getting etcd3 support added to tooz and making sure
>> there is a good functional test for etcd3.
>
>
> We should also make it clear that:
>
> 1) Tokyo was nearly 1.5 years ago.
> 2) Many stake holders in openstack with people in that room may no
> longer be part of our community
> 3) Alignment with Kubernetes has become something important at many
> levels inside of OpenStack (which puts some real weight on the etcd front)


 Yes, and even more so for etcd3 vs. etcd2, since a) k8s now uses etcd3 and
 b) etcd2 is no longer being worked on.

> 4) The containers ecosystem, which etcd came out of, has matured
> dramatically
>>>
>>> +1 for working towards etcd3 a "base service", based on operator acceptance.
>>> +1 for liveness checks not causing silly DB churn.
>>>
>>> While we might not need/want an abstraction layer to hide the
>>> differences between different backends, but a library (tooz and/or
>>> os-lively) so we all consistently use the tool seems to make sense.
>>>
>>> Maybe that means get tooz using etcd3 (Julian or Jay, or both maybe
>>> seemed keen?)
>>> Maybe the tooz API adds bits from the os-lively POC?
>>
>> I do have a concern where we immediately jump to a generic abstraction,
>> instead of using the underlying technology to the best of our ability.
>> It's really hard to break down and optimize the abstractions later.
>> We've got all sorts of cruft (an inefficiencies) in our DB access layer
>> because of this (UUIDs stored as UTF8 strings being a good example).
>>
>> I'd definitely be more interested in etcd3 as a defined base service,
>> people can use it directly. See what kind of patterns people come up
>> with. Abstract late once the patterns are there.
> 
> Good point.
> 
> +1 to collecting the patterns.
> Thats the bit I didn't want to throw away.

++

If the 1.5 years has changed us, I think just depending on etcd3 would
be awesome.

FWIW - when we added zk to zuul/nodepool, we chose a single system
(although the opposite system, but that's not the interesting thing
here) for all the reasons mentioned about being able to dive deep into
the actual tool. It has worked well for us. However, we did wind up
writing an in-tree API on top of kazoo in-tree almost immediately, just
because it made working with it for the logical operations we needed
more sense elsewhere in the code.

So - yeah - use the tech without a generic abstraction - but patterns
can help a bunch.


__
OpenStack Development Mailing List (not 

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Monty Taylor]

On 03/15/2017 11:37 AM, Davanum Srinivas wrote:
> Monty, Team,
> 
> Sorry for the top post:
> 
> Support for etcd/tooz in devstack (with file driver as default) -
> https://review.openstack.org/#/c/445432/
> 
> As of right now both zookeeper driver and etcd driver is working fine:
> https://review.openstack.org/#/c/445630/
> https://review.openstack.org/#/c/445629/
> 
> The problem we have from before is that we do not have any CI jobs
> that used zookeeper.
> 
> I am leaning towards just throwing the etcd as default and if folks
> are interested in zookeeper then they can add specific CI jobs with
> DLM_BACKEND variable set.

That doesn't bother me - zk as the default choice was because at the
time zk worked and etcd did not.

That said - etcd3 is a newer/better thing - so maybe instead of driving
etcd home as a default before we add etcd3 support, we just change tooz
to support etcd3, add the devstack jobs to use that, and start from a
position that doesn't involve dealing with any legacy?

> On Tue, Mar 14, 2017 at 11:00 PM, Monty Taylor  wrote:
>> On 03/15/2017 03:13 AM, Jay Pipes wrote:
>>> On 03/14/2017 05:01 PM, Clint Byrum wrote:
 Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
> On 03/14/2017 02:50 PM, Julien Danjou wrote:
>> On Tue, Mar 14 2017, Jay Pipes wrote:
>>
>>> Not tooz, because I'm not interested in a DLM nor leader election
>>> library
>>> (that's what the underlying etcd3 cluster handles for me), only a
>>> fast service
>>> liveness/healthcheck system, but it shows usage of etcd3 and Google
>>> Protocol
>>> Buffers implementing a simple API for liveness checking and host
>>> maintenance
>>> reporting.
>>
>> Cool cool. So that's the same feature that we implemented in tooz 3
>> years ago. It's called "group membership". You create a group, make
>> nodes join it, and you know who's dead/alive and get notified when
>> their
>> status change.
>
> The point of os-lively is not to provide a thin API over ZooKeeper's
> group membership interface. The point of os-lively is to remove the need
> to have a database (RDBMS) record of a service in Nova.

 That's also the point of tooz's group membership API:

 https://docs.openstack.org/developer/tooz/compatibility.html#grouping
>>>
>>> Did you take a look at the code I wrote in os-lively? What part of the
>>> tooz group membership API do you think I would have used?
>>>
>>> Again, this was a weekend project that I was moving fast on. I looked at
>>> tooz and didn't see how I could use it for my purposes, which was to
>>> store a versioned object in a consistent key/value store with support
>>> for transactional semantics when storing index and data records at the
>>> same time [1]
>>>
>>> https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511
>>>
>>>
>>> etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional
>>> semantics in a consistent key/value store that I needed.
>>>
>>> tooz is cool, but it's not what I was looking for. It's solving a
>>> different problem than I was trying to solve.
>>>
>>> This isn't a case of NIH, despite what Julien is trying to intimate in
>>> his emails.
>>>
> tooz simply abstracts a group membership API across a number of drivers.
> I don't need that. I need a way to maintain a service record (with
> maintenance period information, region, and an evolvable data record
> format) and query those service records in an RDBMS-like manner but
> without the RDBMS being involved.
>
>>> servicegroup API with os-lively and eliminate Nova's use of an
>>> RDBMS for
>>> service liveness checking, which should dramatically reduce the
>>> amount of both
>>> DB traffic as well as conductor/MQ service update traffic.
>>
>> Interesting. Joshua and Vilob tried to push usage of tooz group
>> membership a couple of years ago, but it got nowhere. Well, no, they
>> got
>> 2 specs written IIRC:
>>
>>
>> https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
>>
>>
>> But then it died for whatever reasons on Nova side.
>
> It died because it didn't actually solve a problem.
>
> The problem is that even if we incorporate tooz, we would still need to
> have a service table in the RDBMS and continue to query it over and over
> again in the scheduler and API nodes.

 Most likely it was designed with hesitance to have a tooz requirement
 to be a source of truth. But it's certainly not a problem for most tooz
 backends to be a source of truth. Certainly not for etcd or ZK, which
 are both designed to be that.

> I want all service information in the same place, and I don't want to
> use an RDBMS for that information. etcd3 provides an ideal place to
> store service 

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Nicolas Trangez]

On Wed, 2017-03-15 at 08:33 -0400, Sean Dague wrote:
> On 03/15/2017 08:10 AM, John Garbutt wrote:
> > On 15 March 2017 at 11:58, Jay Pipes  wrote:
> > > On 03/15/2017 07:44 AM, Sean Dague wrote:
> > > > 
> > > > On 03/14/2017 11:00 PM, Monty Taylor wrote:
> > > > 
> > > > > 
> > > > > a) awesome. when the rest of this dips momentarily into words
> > > > > that might
> > > > > sound negative, please hear it all wrapped in an "awesome"
> > > > > and know that
> > > > > my personal desire is to see the thing you're working on be
> > > > > successful
> > > > > without undue burden...
> > > > > 
> > > > > b) In Tokyo, we had the big discussion about DLMs (where at
> > > > > least my
> > > > > intent going in to the room was to get us to pick one and
> > > > > only one).
> > > > > There were three camps in the room who were all vocal:
> > > > > 
> > > > > 1) YES! Let's just pick one, I don't care which one
> > > > > 2) I hate Java I don't want to run Zookeeper, so we can't
> > > > > pick that
> > > > > 3) I hate go/don't trust coreos I don't want to run etcd so
> > > > > we can't
> > > > > pick that
> > > > > 
> > > > > Because of 2 and 3 the group represented by 1 lost and we
> > > > > ended up with:
> > > > > "crap, we have to use an abstraction library"
> > > > > 
> > > > > I'd argue that unless something has changed significantly,
> > > > > having Nova
> > > > > grow a direct depend on etcd when the DLM discussion brought
> > > > > us to "the
> > > > > operators in the room have expressed a need for a pluggable
> > > > > choice
> > > > > between at least zk and etcd" should be pretty much a non-
> > > > > starter.
> > > > > 
> > > > > Now, being that I was personally in group 1, I'd be THRILLED
> > > > > if we
> > > > > could, as a community, decide to pick one and skip having an
> > > > > abstraction
> > > > > library. I still don't care which one - and you know I love
> > > > > gRPC/protobuf.
> > > > > 
> > > > > But I do think that given the anti-etcd sentiment that was
> > > > > expressed was
> > > > > equally as vehement as the anti-zk sentiment, that we need to
> > > > > circle
> > > > > back and make a legit call on this topic.
> > > > > 
> > > > > If we can pick one, I think having special-purpose libraries
> > > > > like
> > > > > os-lively for specific purposes would be neat.
> > > > > 
> > > > > If we still can't pick one, then I think adding the liveness
> > > > > check you
> > > > > implemented for os-lively as a new feature in tooz and also
> > > > > implementing
> > > > > the same thing in the zk driver would be necessary. (of
> > > > > course, that'll
> > > > > probably depend on getting etcd3 support added to tooz and
> > > > > making sure
> > > > > there is a good functional test for etcd3.
> > > > 
> > > > 
> > > > We should also make it clear that:
> > > > 
> > > > 1) Tokyo was nearly 1.5 years ago.
> > > > 2) Many stake holders in openstack with people in that room may
> > > > no
> > > > longer be part of our community
> > > > 3) Alignment with Kubernetes has become something important at
> > > > many
> > > > levels inside of OpenStack (which puts some real weight on the
> > > > etcd front)
> > > 
> > > 
> > > Yes, and even more so for etcd3 vs. etcd2, since a) k8s now uses
> > > etcd3 and
> > > b) etcd2 is no longer being worked on.
> > > 
> > > > 4) The containers ecosystem, which etcd came out of, has
> > > > matured
> > > > dramatically
> > 
> > +1 for working towards etcd3 a "base service", based on operator
> > acceptance.
> > +1 for liveness checks not causing silly DB churn.
> > 
> > While we might not need/want an abstraction layer to hide the
> > differences between different backends, but a library (tooz and/or
> > os-lively) so we all consistently use the tool seems to make sense.
> > 
> > Maybe that means get tooz using etcd3 (Julian or Jay, or both maybe
> > seemed keen?)
> > Maybe the tooz API adds bits from the os-lively POC?
> 
> I do have a concern where we immediately jump to a generic
> abstraction,
> instead of using the underlying technology to the best of our
> ability.
> It's really hard to break down and optimize the abstractions later.
> We've got all sorts of cruft (an inefficiencies) in our DB access
> layer
> because of this (UUIDs stored as UTF8 strings being a good example).
> 
> I'd definitely be more interested in etcd3 as a defined base service,
> people can use it directly. See what kind of patterns people come up
> with. Abstract late once the patterns are there.

I'm not involved in Oslo or Devstack, so not really a say in this, but:

Yes! A 1000 times yes! An 'abstraction' with only a single
implementation is (1) almost by definition not an abstraction, and (2)
in 99% of the cases just exposing a crippled version of the underlying
system's features.
Furthermore, adding a second implementation of the interface backed by
another system at some later point in time turns out to be difficult
(or impossible) because the 'abstraction' 

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[John Garbutt]

On 15 March 2017 at 12:33, Sean Dague  wrote:
> On 03/15/2017 08:10 AM, John Garbutt wrote:
>> On 15 March 2017 at 11:58, Jay Pipes  wrote:
>>> On 03/15/2017 07:44 AM, Sean Dague wrote:

 On 03/14/2017 11:00 PM, Monty Taylor wrote:
 
>
> a) awesome. when the rest of this dips momentarily into words that might
> sound negative, please hear it all wrapped in an "awesome" and know that
> my personal desire is to see the thing you're working on be successful
> without undue burden...
>
> b) In Tokyo, we had the big discussion about DLMs (where at least my
> intent going in to the room was to get us to pick one and only one).
> There were three camps in the room who were all vocal:
>
> 1) YES! Let's just pick one, I don't care which one
> 2) I hate Java I don't want to run Zookeeper, so we can't pick that
> 3) I hate go/don't trust coreos I don't want to run etcd so we can't
> pick that
>
> Because of 2 and 3 the group represented by 1 lost and we ended up with:
> "crap, we have to use an abstraction library"
>
> I'd argue that unless something has changed significantly, having Nova
> grow a direct depend on etcd when the DLM discussion brought us to "the
> operators in the room have expressed a need for a pluggable choice
> between at least zk and etcd" should be pretty much a non-starter.
>
> Now, being that I was personally in group 1, I'd be THRILLED if we
> could, as a community, decide to pick one and skip having an abstraction
> library. I still don't care which one - and you know I love
> gRPC/protobuf.
>
> But I do think that given the anti-etcd sentiment that was expressed was
> equally as vehement as the anti-zk sentiment, that we need to circle
> back and make a legit call on this topic.
>
> If we can pick one, I think having special-purpose libraries like
> os-lively for specific purposes would be neat.
>
> If we still can't pick one, then I think adding the liveness check you
> implemented for os-lively as a new feature in tooz and also implementing
> the same thing in the zk driver would be necessary. (of course, that'll
> probably depend on getting etcd3 support added to tooz and making sure
> there is a good functional test for etcd3.


 We should also make it clear that:

 1) Tokyo was nearly 1.5 years ago.
 2) Many stake holders in openstack with people in that room may no
 longer be part of our community
 3) Alignment with Kubernetes has become something important at many
 levels inside of OpenStack (which puts some real weight on the etcd front)
>>>
>>>
>>> Yes, and even more so for etcd3 vs. etcd2, since a) k8s now uses etcd3 and
>>> b) etcd2 is no longer being worked on.
>>>
 4) The containers ecosystem, which etcd came out of, has matured
 dramatically
>>
>> +1 for working towards etcd3 a "base service", based on operator acceptance.
>> +1 for liveness checks not causing silly DB churn.
>>
>> While we might not need/want an abstraction layer to hide the
>> differences between different backends, but a library (tooz and/or
>> os-lively) so we all consistently use the tool seems to make sense.
>>
>> Maybe that means get tooz using etcd3 (Julian or Jay, or both maybe
>> seemed keen?)
>> Maybe the tooz API adds bits from the os-lively POC?
>
> I do have a concern where we immediately jump to a generic abstraction,
> instead of using the underlying technology to the best of our ability.
> It's really hard to break down and optimize the abstractions later.
> We've got all sorts of cruft (an inefficiencies) in our DB access layer
> because of this (UUIDs stored as UTF8 strings being a good example).
>
> I'd definitely be more interested in etcd3 as a defined base service,
> people can use it directly. See what kind of patterns people come up
> with. Abstract late once the patterns are there.

Good point.

+1 to collecting the patterns.
Thats the bit I didn't want to throw away.

Thanks,
John

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Sean Dague]

On 03/15/2017 08:10 AM, John Garbutt wrote:
> On 15 March 2017 at 11:58, Jay Pipes  wrote:
>> On 03/15/2017 07:44 AM, Sean Dague wrote:
>>>
>>> On 03/14/2017 11:00 PM, Monty Taylor wrote:
>>> 

 a) awesome. when the rest of this dips momentarily into words that might
 sound negative, please hear it all wrapped in an "awesome" and know that
 my personal desire is to see the thing you're working on be successful
 without undue burden...

 b) In Tokyo, we had the big discussion about DLMs (where at least my
 intent going in to the room was to get us to pick one and only one).
 There were three camps in the room who were all vocal:

 1) YES! Let's just pick one, I don't care which one
 2) I hate Java I don't want to run Zookeeper, so we can't pick that
 3) I hate go/don't trust coreos I don't want to run etcd so we can't
 pick that

 Because of 2 and 3 the group represented by 1 lost and we ended up with:
 "crap, we have to use an abstraction library"

 I'd argue that unless something has changed significantly, having Nova
 grow a direct depend on etcd when the DLM discussion brought us to "the
 operators in the room have expressed a need for a pluggable choice
 between at least zk and etcd" should be pretty much a non-starter.

 Now, being that I was personally in group 1, I'd be THRILLED if we
 could, as a community, decide to pick one and skip having an abstraction
 library. I still don't care which one - and you know I love
 gRPC/protobuf.

 But I do think that given the anti-etcd sentiment that was expressed was
 equally as vehement as the anti-zk sentiment, that we need to circle
 back and make a legit call on this topic.

 If we can pick one, I think having special-purpose libraries like
 os-lively for specific purposes would be neat.

 If we still can't pick one, then I think adding the liveness check you
 implemented for os-lively as a new feature in tooz and also implementing
 the same thing in the zk driver would be necessary. (of course, that'll
 probably depend on getting etcd3 support added to tooz and making sure
 there is a good functional test for etcd3.
>>>
>>>
>>> We should also make it clear that:
>>>
>>> 1) Tokyo was nearly 1.5 years ago.
>>> 2) Many stake holders in openstack with people in that room may no
>>> longer be part of our community
>>> 3) Alignment with Kubernetes has become something important at many
>>> levels inside of OpenStack (which puts some real weight on the etcd front)
>>
>>
>> Yes, and even more so for etcd3 vs. etcd2, since a) k8s now uses etcd3 and
>> b) etcd2 is no longer being worked on.
>>
>>> 4) The containers ecosystem, which etcd came out of, has matured
>>> dramatically
> 
> +1 for working towards etcd3 a "base service", based on operator acceptance.
> +1 for liveness checks not causing silly DB churn.
> 
> While we might not need/want an abstraction layer to hide the
> differences between different backends, but a library (tooz and/or
> os-lively) so we all consistently use the tool seems to make sense.
> 
> Maybe that means get tooz using etcd3 (Julian or Jay, or both maybe
> seemed keen?)
> Maybe the tooz API adds bits from the os-lively POC?

I do have a concern where we immediately jump to a generic abstraction,
instead of using the underlying technology to the best of our ability.
It's really hard to break down and optimize the abstractions later.
We've got all sorts of cruft (an inefficiencies) in our DB access layer
because of this (UUIDs stored as UTF8 strings being a good example).

I'd definitely be more interested in etcd3 as a defined base service,
people can use it directly. See what kind of patterns people come up
with. Abstract late once the patterns are there.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[John Garbutt]

On 15 March 2017 at 11:58, Jay Pipes  wrote:
> On 03/15/2017 07:44 AM, Sean Dague wrote:
>>
>> On 03/14/2017 11:00 PM, Monty Taylor wrote:
>> 
>>>
>>> a) awesome. when the rest of this dips momentarily into words that might
>>> sound negative, please hear it all wrapped in an "awesome" and know that
>>> my personal desire is to see the thing you're working on be successful
>>> without undue burden...
>>>
>>> b) In Tokyo, we had the big discussion about DLMs (where at least my
>>> intent going in to the room was to get us to pick one and only one).
>>> There were three camps in the room who were all vocal:
>>>
>>> 1) YES! Let's just pick one, I don't care which one
>>> 2) I hate Java I don't want to run Zookeeper, so we can't pick that
>>> 3) I hate go/don't trust coreos I don't want to run etcd so we can't
>>> pick that
>>>
>>> Because of 2 and 3 the group represented by 1 lost and we ended up with:
>>> "crap, we have to use an abstraction library"
>>>
>>> I'd argue that unless something has changed significantly, having Nova
>>> grow a direct depend on etcd when the DLM discussion brought us to "the
>>> operators in the room have expressed a need for a pluggable choice
>>> between at least zk and etcd" should be pretty much a non-starter.
>>>
>>> Now, being that I was personally in group 1, I'd be THRILLED if we
>>> could, as a community, decide to pick one and skip having an abstraction
>>> library. I still don't care which one - and you know I love
>>> gRPC/protobuf.
>>>
>>> But I do think that given the anti-etcd sentiment that was expressed was
>>> equally as vehement as the anti-zk sentiment, that we need to circle
>>> back and make a legit call on this topic.
>>>
>>> If we can pick one, I think having special-purpose libraries like
>>> os-lively for specific purposes would be neat.
>>>
>>> If we still can't pick one, then I think adding the liveness check you
>>> implemented for os-lively as a new feature in tooz and also implementing
>>> the same thing in the zk driver would be necessary. (of course, that'll
>>> probably depend on getting etcd3 support added to tooz and making sure
>>> there is a good functional test for etcd3.
>>
>>
>> We should also make it clear that:
>>
>> 1) Tokyo was nearly 1.5 years ago.
>> 2) Many stake holders in openstack with people in that room may no
>> longer be part of our community
>> 3) Alignment with Kubernetes has become something important at many
>> levels inside of OpenStack (which puts some real weight on the etcd front)
>
>
> Yes, and even more so for etcd3 vs. etcd2, since a) k8s now uses etcd3 and
> b) etcd2 is no longer being worked on.
>
>> 4) The containers ecosystem, which etcd came out of, has matured
>> dramatically

+1 for working towards etcd3 a "base service", based on operator acceptance.
+1 for liveness checks not causing silly DB churn.

While we might not need/want an abstraction layer to hide the
differences between different backends, but a library (tooz and/or
os-lively) so we all consistently use the tool seems to make sense.

Maybe that means get tooz using etcd3 (Julian or Jay, or both maybe
seemed keen?)
Maybe the tooz API adds bits from the os-lively POC?

John

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Jay Pipes]

On 03/15/2017 07:44 AM, Sean Dague wrote:

On 03/14/2017 11:00 PM, Monty Taylor wrote:


a) awesome. when the rest of this dips momentarily into words that might
sound negative, please hear it all wrapped in an "awesome" and know that
my personal desire is to see the thing you're working on be successful
without undue burden...

b) In Tokyo, we had the big discussion about DLMs (where at least my
intent going in to the room was to get us to pick one and only one).
There were three camps in the room who were all vocal:

1) YES! Let's just pick one, I don't care which one
2) I hate Java I don't want to run Zookeeper, so we can't pick that
3) I hate go/don't trust coreos I don't want to run etcd so we can't
pick that

Because of 2 and 3 the group represented by 1 lost and we ended up with:
"crap, we have to use an abstraction library"

I'd argue that unless something has changed significantly, having Nova
grow a direct depend on etcd when the DLM discussion brought us to "the
operators in the room have expressed a need for a pluggable choice
between at least zk and etcd" should be pretty much a non-starter.

Now, being that I was personally in group 1, I'd be THRILLED if we
could, as a community, decide to pick one and skip having an abstraction
library. I still don't care which one - and you know I love gRPC/protobuf.

But I do think that given the anti-etcd sentiment that was expressed was
equally as vehement as the anti-zk sentiment, that we need to circle
back and make a legit call on this topic.

If we can pick one, I think having special-purpose libraries like
os-lively for specific purposes would be neat.

If we still can't pick one, then I think adding the liveness check you
implemented for os-lively as a new feature in tooz and also implementing
the same thing in the zk driver would be necessary. (of course, that'll
probably depend on getting etcd3 support added to tooz and making sure
there is a good functional test for etcd3.


We should also make it clear that:

1) Tokyo was nearly 1.5 years ago.
2) Many stake holders in openstack with people in that room may no
longer be part of our community
3) Alignment with Kubernetes has become something important at many
levels inside of OpenStack (which puts some real weight on the etcd front)


Yes, and even more so for etcd3 vs. etcd2, since a) k8s now uses etcd3 
and b) etcd2 is no longer being worked on.



4) The containers ecosystem, which etcd came out of, has matured
dramatically


Some of it has matured, yes :) Other parts of it continue to be a 
whirling maelstrom of unpredictability^Winnovation.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Davanum Srinivas]

+1000 Sean.

Thanks,
Dims

On Wed, Mar 15, 2017 at 7:44 AM, Sean Dague  wrote:
> On 03/14/2017 11:00 PM, Monty Taylor wrote:
> 
>> a) awesome. when the rest of this dips momentarily into words that might
>> sound negative, please hear it all wrapped in an "awesome" and know that
>> my personal desire is to see the thing you're working on be successful
>> without undue burden...
>>
>> b) In Tokyo, we had the big discussion about DLMs (where at least my
>> intent going in to the room was to get us to pick one and only one).
>> There were three camps in the room who were all vocal:
>>
>> 1) YES! Let's just pick one, I don't care which one
>> 2) I hate Java I don't want to run Zookeeper, so we can't pick that
>> 3) I hate go/don't trust coreos I don't want to run etcd so we can't
>> pick that
>>
>> Because of 2 and 3 the group represented by 1 lost and we ended up with:
>> "crap, we have to use an abstraction library"
>>
>> I'd argue that unless something has changed significantly, having Nova
>> grow a direct depend on etcd when the DLM discussion brought us to "the
>> operators in the room have expressed a need for a pluggable choice
>> between at least zk and etcd" should be pretty much a non-starter.
>>
>> Now, being that I was personally in group 1, I'd be THRILLED if we
>> could, as a community, decide to pick one and skip having an abstraction
>> library. I still don't care which one - and you know I love gRPC/protobuf.
>>
>> But I do think that given the anti-etcd sentiment that was expressed was
>> equally as vehement as the anti-zk sentiment, that we need to circle
>> back and make a legit call on this topic.
>>
>> If we can pick one, I think having special-purpose libraries like
>> os-lively for specific purposes would be neat.
>>
>> If we still can't pick one, then I think adding the liveness check you
>> implemented for os-lively as a new feature in tooz and also implementing
>> the same thing in the zk driver would be necessary. (of course, that'll
>> probably depend on getting etcd3 support added to tooz and making sure
>> there is a good functional test for etcd3.
>
> We should also make it clear that:
>
> 1) Tokyo was nearly 1.5 years ago.
> 2) Many stake holders in openstack with people in that room may no
> longer be part of our community
> 3) Alignment with Kubernetes has become something important at many
> levels inside of OpenStack (which puts some real weight on the etcd front)
> 4) The containers ecosystem, which etcd came out of, has matured
> dramatically
>
> I do think this is enough change to when that decision was made to
> revisit. As was said elsewhere in this thread, you have to run etcd for
> kubernetes, so picking that (in an opinionated way) for OpenStack seems
> like a good both technical and social call.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Sean Dague]

On 03/14/2017 11:00 PM, Monty Taylor wrote:

> a) awesome. when the rest of this dips momentarily into words that might
> sound negative, please hear it all wrapped in an "awesome" and know that
> my personal desire is to see the thing you're working on be successful
> without undue burden...
> 
> b) In Tokyo, we had the big discussion about DLMs (where at least my
> intent going in to the room was to get us to pick one and only one).
> There were three camps in the room who were all vocal:
> 
> 1) YES! Let's just pick one, I don't care which one
> 2) I hate Java I don't want to run Zookeeper, so we can't pick that
> 3) I hate go/don't trust coreos I don't want to run etcd so we can't
> pick that
> 
> Because of 2 and 3 the group represented by 1 lost and we ended up with:
> "crap, we have to use an abstraction library"
> 
> I'd argue that unless something has changed significantly, having Nova
> grow a direct depend on etcd when the DLM discussion brought us to "the
> operators in the room have expressed a need for a pluggable choice
> between at least zk and etcd" should be pretty much a non-starter.
> 
> Now, being that I was personally in group 1, I'd be THRILLED if we
> could, as a community, decide to pick one and skip having an abstraction
> library. I still don't care which one - and you know I love gRPC/protobuf.
> 
> But I do think that given the anti-etcd sentiment that was expressed was
> equally as vehement as the anti-zk sentiment, that we need to circle
> back and make a legit call on this topic.
> 
> If we can pick one, I think having special-purpose libraries like
> os-lively for specific purposes would be neat.
> 
> If we still can't pick one, then I think adding the liveness check you
> implemented for os-lively as a new feature in tooz and also implementing
> the same thing in the zk driver would be necessary. (of course, that'll
> probably depend on getting etcd3 support added to tooz and making sure
> there is a good functional test for etcd3.

We should also make it clear that:

1) Tokyo was nearly 1.5 years ago.
2) Many stake holders in openstack with people in that room may no
longer be part of our community
3) Alignment with Kubernetes has become something important at many
levels inside of OpenStack (which puts some real weight on the etcd front)
4) The containers ecosystem, which etcd came out of, has matured
dramatically

I do think this is enough change to when that decision was made to
revisit. As was said elsewhere in this thread, you have to run etcd for
kubernetes, so picking that (in an opinionated way) for OpenStack seems
like a good both technical and social call.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Davanum Srinivas]

Monty, Team,

Sorry for the top post:

Support for etcd/tooz in devstack (with file driver as default) -
https://review.openstack.org/#/c/445432/

As of right now both zookeeper driver and etcd driver is working fine:
https://review.openstack.org/#/c/445630/
https://review.openstack.org/#/c/445629/

The problem we have from before is that we do not have any CI jobs
that used zookeeper.

I am leaning towards just throwing the etcd as default and if folks
are interested in zookeeper then they can add specific CI jobs with
DLM_BACKEND variable set.

Thanks,
Dims

On Tue, Mar 14, 2017 at 11:00 PM, Monty Taylor  wrote:
> On 03/15/2017 03:13 AM, Jay Pipes wrote:
>> On 03/14/2017 05:01 PM, Clint Byrum wrote:
>>> Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
 On 03/14/2017 02:50 PM, Julien Danjou wrote:
> On Tue, Mar 14 2017, Jay Pipes wrote:
>
>> Not tooz, because I'm not interested in a DLM nor leader election
>> library
>> (that's what the underlying etcd3 cluster handles for me), only a
>> fast service
>> liveness/healthcheck system, but it shows usage of etcd3 and Google
>> Protocol
>> Buffers implementing a simple API for liveness checking and host
>> maintenance
>> reporting.
>
> Cool cool. So that's the same feature that we implemented in tooz 3
> years ago. It's called "group membership". You create a group, make
> nodes join it, and you know who's dead/alive and get notified when
> their
> status change.

 The point of os-lively is not to provide a thin API over ZooKeeper's
 group membership interface. The point of os-lively is to remove the need
 to have a database (RDBMS) record of a service in Nova.
>>>
>>> That's also the point of tooz's group membership API:
>>>
>>> https://docs.openstack.org/developer/tooz/compatibility.html#grouping
>>
>> Did you take a look at the code I wrote in os-lively? What part of the
>> tooz group membership API do you think I would have used?
>>
>> Again, this was a weekend project that I was moving fast on. I looked at
>> tooz and didn't see how I could use it for my purposes, which was to
>> store a versioned object in a consistent key/value store with support
>> for transactional semantics when storing index and data records at the
>> same time [1]
>>
>> https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511
>>
>>
>> etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional
>> semantics in a consistent key/value store that I needed.
>>
>> tooz is cool, but it's not what I was looking for. It's solving a
>> different problem than I was trying to solve.
>>
>> This isn't a case of NIH, despite what Julien is trying to intimate in
>> his emails.
>>
 tooz simply abstracts a group membership API across a number of drivers.
 I don't need that. I need a way to maintain a service record (with
 maintenance period information, region, and an evolvable data record
 format) and query those service records in an RDBMS-like manner but
 without the RDBMS being involved.

>> servicegroup API with os-lively and eliminate Nova's use of an
>> RDBMS for
>> service liveness checking, which should dramatically reduce the
>> amount of both
>> DB traffic as well as conductor/MQ service update traffic.
>
> Interesting. Joshua and Vilob tried to push usage of tooz group
> membership a couple of years ago, but it got nowhere. Well, no, they
> got
> 2 specs written IIRC:
>
>
> https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
>
>
> But then it died for whatever reasons on Nova side.

 It died because it didn't actually solve a problem.

 The problem is that even if we incorporate tooz, we would still need to
 have a service table in the RDBMS and continue to query it over and over
 again in the scheduler and API nodes.
>>>
>>> Most likely it was designed with hesitance to have a tooz requirement
>>> to be a source of truth. But it's certainly not a problem for most tooz
>>> backends to be a source of truth. Certainly not for etcd or ZK, which
>>> are both designed to be that.
>>>
 I want all service information in the same place, and I don't want to
 use an RDBMS for that information. etcd3 provides an ideal place to
 store service record information. Google Protocol Buffers is an ideal
 data format for evolvable versioned objects. os-lively presents an API
 that solves the problem I want to solve in Nova. tooz didn't.
>>>
>>> Was there something inherent in tooz's design that prevented you from
>>> adding it to tooz's group API? Said API already includes liveness (watch
>>> the group that corresponds to the service you want).
>>
>> See above about transactional semantics.
>>
>> I'm actually happy to add an etcd3 group membership driver to tooz,
>> 

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Clint Byrum]

Excerpts from Jay Pipes's message of 2017-03-14 22:13:32 -0400:
> On 03/14/2017 05:01 PM, Clint Byrum wrote:
> > Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
> >> On 03/14/2017 02:50 PM, Julien Danjou wrote:
> >>> On Tue, Mar 14 2017, Jay Pipes wrote:
> >>>
>  Not tooz, because I'm not interested in a DLM nor leader election library
>  (that's what the underlying etcd3 cluster handles for me), only a fast 
>  service
>  liveness/healthcheck system, but it shows usage of etcd3 and Google 
>  Protocol
>  Buffers implementing a simple API for liveness checking and host 
>  maintenance
>  reporting.
> >>>
> >>> Cool cool. So that's the same feature that we implemented in tooz 3
> >>> years ago. It's called "group membership". You create a group, make
> >>> nodes join it, and you know who's dead/alive and get notified when their
> >>> status change.
> >>
> >> The point of os-lively is not to provide a thin API over ZooKeeper's
> >> group membership interface. The point of os-lively is to remove the need
> >> to have a database (RDBMS) record of a service in Nova.
> >
> > That's also the point of tooz's group membership API:
> >
> > https://docs.openstack.org/developer/tooz/compatibility.html#grouping
> 
> Did you take a look at the code I wrote in os-lively? What part of the 
> tooz group membership API do you think I would have used?
> 
> Again, this was a weekend project that I was moving fast on. I looked at 
> tooz and didn't see how I could use it for my purposes, which was to 
> store a versioned object in a consistent key/value store with support 
> for transactional semantics when storing index and data records at the 
> same time [1]
> 
> https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511
> 
> etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional 
> semantics in a consistent key/value store that I needed.
> 

ZK has all the primitives necessary, and the client libs behind tooz for
it have transaction support basically identical to etcd3's:

https://kazoo.readthedocs.io/en/latest/_modules/kazoo/client.html#KazooClient.transaction

> tooz is cool, but it's not what I was looking for. It's solving a 
> different problem than I was trying to solve.
> 
> This isn't a case of NIH, despite what Julien is trying to intimate in 
> his emails.
> 

Yeah I don't want to imply that either. I'm trying to figure out how we
can add what you're doing to tooz, not why you didn't see something or
why you reinvented something.

> >> tooz simply abstracts a group membership API across a number of drivers.
> >> I don't need that. I need a way to maintain a service record (with
> >> maintenance period information, region, and an evolvable data record
> >> format) and query those service records in an RDBMS-like manner but
> >> without the RDBMS being involved.
> >>
>  servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
>  service liveness checking, which should dramatically reduce the amount 
>  of both
>  DB traffic as well as conductor/MQ service update traffic.
> >>>
> >>> Interesting. Joshua and Vilob tried to push usage of tooz group
> >>> membership a couple of years ago, but it got nowhere. Well, no, they got
> >>> 2 specs written IIRC:
> >>>
> >>>   
> >>> https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
> >>>
> >>> But then it died for whatever reasons on Nova side.
> >>
> >> It died because it didn't actually solve a problem.
> >>
> >> The problem is that even if we incorporate tooz, we would still need to
> >> have a service table in the RDBMS and continue to query it over and over
> >> again in the scheduler and API nodes.
> >
> > Most likely it was designed with hesitance to have a tooz requirement
> > to be a source of truth. But it's certainly not a problem for most tooz
> > backends to be a source of truth. Certainly not for etcd or ZK, which
> > are both designed to be that.
> >
> >> I want all service information in the same place, and I don't want to
> >> use an RDBMS for that information. etcd3 provides an ideal place to
> >> store service record information. Google Protocol Buffers is an ideal
> >> data format for evolvable versioned objects. os-lively presents an API
> >> that solves the problem I want to solve in Nova. tooz didn't.
> >
> > Was there something inherent in tooz's design that prevented you from
> > adding it to tooz's group API? Said API already includes liveness (watch
> > the group that corresponds to the service you want).
> 
> See above about transactional semantics.
> 
> I'm actually happy to add an etcd3 group membership driver to tooz, 
> though. After the experience gained this weekend using etcd3, I'd like 
> to do that.
> 

\o/

> Still doesn't mean that tooz would be the appropriate choice for what I 
> was trying to do with os-lively, though.
> 

Monty makes a pretty strong case for hiding 

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Monty Taylor]

On 03/15/2017 03:13 AM, Jay Pipes wrote:
> On 03/14/2017 05:01 PM, Clint Byrum wrote:
>> Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
>>> On 03/14/2017 02:50 PM, Julien Danjou wrote:
 On Tue, Mar 14 2017, Jay Pipes wrote:

> Not tooz, because I'm not interested in a DLM nor leader election
> library
> (that's what the underlying etcd3 cluster handles for me), only a
> fast service
> liveness/healthcheck system, but it shows usage of etcd3 and Google
> Protocol
> Buffers implementing a simple API for liveness checking and host
> maintenance
> reporting.

 Cool cool. So that's the same feature that we implemented in tooz 3
 years ago. It's called "group membership". You create a group, make
 nodes join it, and you know who's dead/alive and get notified when
 their
 status change.
>>>
>>> The point of os-lively is not to provide a thin API over ZooKeeper's
>>> group membership interface. The point of os-lively is to remove the need
>>> to have a database (RDBMS) record of a service in Nova.
>>
>> That's also the point of tooz's group membership API:
>>
>> https://docs.openstack.org/developer/tooz/compatibility.html#grouping
> 
> Did you take a look at the code I wrote in os-lively? What part of the
> tooz group membership API do you think I would have used?
> 
> Again, this was a weekend project that I was moving fast on. I looked at
> tooz and didn't see how I could use it for my purposes, which was to
> store a versioned object in a consistent key/value store with support
> for transactional semantics when storing index and data records at the
> same time [1]
> 
> https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511
> 
> 
> etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional
> semantics in a consistent key/value store that I needed.
> 
> tooz is cool, but it's not what I was looking for. It's solving a
> different problem than I was trying to solve.
> 
> This isn't a case of NIH, despite what Julien is trying to intimate in
> his emails.
> 
>>> tooz simply abstracts a group membership API across a number of drivers.
>>> I don't need that. I need a way to maintain a service record (with
>>> maintenance period information, region, and an evolvable data record
>>> format) and query those service records in an RDBMS-like manner but
>>> without the RDBMS being involved.
>>>
> servicegroup API with os-lively and eliminate Nova's use of an
> RDBMS for
> service liveness checking, which should dramatically reduce the
> amount of both
> DB traffic as well as conductor/MQ service update traffic.

 Interesting. Joshua and Vilob tried to push usage of tooz group
 membership a couple of years ago, but it got nowhere. Well, no, they
 got
 2 specs written IIRC:

  
 https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html


 But then it died for whatever reasons on Nova side.
>>>
>>> It died because it didn't actually solve a problem.
>>>
>>> The problem is that even if we incorporate tooz, we would still need to
>>> have a service table in the RDBMS and continue to query it over and over
>>> again in the scheduler and API nodes.
>>
>> Most likely it was designed with hesitance to have a tooz requirement
>> to be a source of truth. But it's certainly not a problem for most tooz
>> backends to be a source of truth. Certainly not for etcd or ZK, which
>> are both designed to be that.
>>
>>> I want all service information in the same place, and I don't want to
>>> use an RDBMS for that information. etcd3 provides an ideal place to
>>> store service record information. Google Protocol Buffers is an ideal
>>> data format for evolvable versioned objects. os-lively presents an API
>>> that solves the problem I want to solve in Nova. tooz didn't.
>>
>> Was there something inherent in tooz's design that prevented you from
>> adding it to tooz's group API? Said API already includes liveness (watch
>> the group that corresponds to the service you want).
> 
> See above about transactional semantics.
> 
> I'm actually happy to add an etcd3 group membership driver to tooz,
> though. After the experience gained this weekend using etcd3, I'd like
> to do that.
> 
> Still doesn't mean that tooz would be the appropriate choice for what I
> was trying to do with os-lively, though.
> 
>> The only thing missing is being able to get groups and group members
>> by secondary indexes. etcd3's built in indexes by field are pretty nice
> 
> Not sure what you're talking about. etcd3 doesn't have any indexing by
> field. I built the os-lively library primarily as a well-defined set of
> index overlays (by uuid, by host, by service type, and by region) over
> etcd3's key/value store.
> 
>> for that, but ZK can likely also do it too by maintaining the index in
>> the driver.
> 
> Maybe, I'm not sure, I didn't 

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Jay Pipes]

On 03/14/2017 05:01 PM, Clint Byrum wrote:

Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:

On 03/14/2017 02:50 PM, Julien Danjou wrote:

On Tue, Mar 14 2017, Jay Pipes wrote:


Not tooz, because I'm not interested in a DLM nor leader election library
(that's what the underlying etcd3 cluster handles for me), only a fast service
liveness/healthcheck system, but it shows usage of etcd3 and Google Protocol
Buffers implementing a simple API for liveness checking and host maintenance
reporting.


Cool cool. So that's the same feature that we implemented in tooz 3
years ago. It's called "group membership". You create a group, make
nodes join it, and you know who's dead/alive and get notified when their
status change.


The point of os-lively is not to provide a thin API over ZooKeeper's
group membership interface. The point of os-lively is to remove the need
to have a database (RDBMS) record of a service in Nova.


That's also the point of tooz's group membership API:

https://docs.openstack.org/developer/tooz/compatibility.html#grouping


Did you take a look at the code I wrote in os-lively? What part of the 
tooz group membership API do you think I would have used?


Again, this was a weekend project that I was moving fast on. I looked at 
tooz and didn't see how I could use it for my purposes, which was to 
store a versioned object in a consistent key/value store with support 
for transactional semantics when storing index and data records at the 
same time [1]


https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511

etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional 
semantics in a consistent key/value store that I needed.


tooz is cool, but it's not what I was looking for. It's solving a 
different problem than I was trying to solve.


This isn't a case of NIH, despite what Julien is trying to intimate in 
his emails.



tooz simply abstracts a group membership API across a number of drivers.
I don't need that. I need a way to maintain a service record (with
maintenance period information, region, and an evolvable data record
format) and query those service records in an RDBMS-like manner but
without the RDBMS being involved.


servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
service liveness checking, which should dramatically reduce the amount of both
DB traffic as well as conductor/MQ service update traffic.


Interesting. Joshua and Vilob tried to push usage of tooz group
membership a couple of years ago, but it got nowhere. Well, no, they got
2 specs written IIRC:

  
https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html

But then it died for whatever reasons on Nova side.


It died because it didn't actually solve a problem.

The problem is that even if we incorporate tooz, we would still need to
have a service table in the RDBMS and continue to query it over and over
again in the scheduler and API nodes.


Most likely it was designed with hesitance to have a tooz requirement
to be a source of truth. But it's certainly not a problem for most tooz
backends to be a source of truth. Certainly not for etcd or ZK, which
are both designed to be that.


I want all service information in the same place, and I don't want to
use an RDBMS for that information. etcd3 provides an ideal place to
store service record information. Google Protocol Buffers is an ideal
data format for evolvable versioned objects. os-lively presents an API
that solves the problem I want to solve in Nova. tooz didn't.


Was there something inherent in tooz's design that prevented you from
adding it to tooz's group API? Said API already includes liveness (watch
the group that corresponds to the service you want).


See above about transactional semantics.

I'm actually happy to add an etcd3 group membership driver to tooz, 
though. After the experience gained this weekend using etcd3, I'd like 
to do that.


Still doesn't mean that tooz would be the appropriate choice for what I 
was trying to do with os-lively, though.



The only thing missing is being able to get groups and group members
by secondary indexes. etcd3's built in indexes by field are pretty nice


Not sure what you're talking about. etcd3 doesn't have any indexing by 
field. I built the os-lively library primarily as a well-defined set of 
index overlays (by uuid, by host, by service type, and by region) over 
etcd3's key/value store.



for that, but ZK can likely also do it too by maintaining the index in
the driver.


Maybe, I'm not sure, I didn't spend much time this weekend looking at 
ZooKeeper.



I understand abstractions can seem pretty cumbersome when you're moving
fast. It's not something I want to see stand in your way. But it would
be nice to see where there's deficiency in tooz so we can be there for
the next project that needs it and maybe eventually factor out direct
etcd3 usage so users who have maybe chosen ZK as 

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Julien Danjou]

On Tue, Mar 14 2017, Clint Byrum wrote:

> I understand abstractions can seem pretty cumbersome when you're moving
> fast.

Considering the problem is at least 5 years old in Nova and tooz itself
is at least 3 years old, let's say that "moving fast" has a… funny
taste.

> It's not something I want to see stand in your way. But it would be
> nice to see where there's deficiency in tooz so we can be there for the
> next project that needs it and maybe eventually factor out direct etcd3
> usage so users who have maybe chosen ZK as their tooz backend can also
> benefit from your work.

+1

-- 
Julien Danjou
/* Free Software hacker
   https://julien.danjou.info */


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Clint Byrum]

Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
> On 03/14/2017 02:50 PM, Julien Danjou wrote:
> > On Tue, Mar 14 2017, Jay Pipes wrote:
> >
> >> Not tooz, because I'm not interested in a DLM nor leader election library
> >> (that's what the underlying etcd3 cluster handles for me), only a fast 
> >> service
> >> liveness/healthcheck system, but it shows usage of etcd3 and Google 
> >> Protocol
> >> Buffers implementing a simple API for liveness checking and host 
> >> maintenance
> >> reporting.
> >
> > Cool cool. So that's the same feature that we implemented in tooz 3
> > years ago. It's called "group membership". You create a group, make
> > nodes join it, and you know who's dead/alive and get notified when their
> > status change.
> 
> The point of os-lively is not to provide a thin API over ZooKeeper's 
> group membership interface. The point of os-lively is to remove the need 
> to have a database (RDBMS) record of a service in Nova.
> 

That's also the point of tooz's group membership API:

https://docs.openstack.org/developer/tooz/compatibility.html#grouping

> tooz simply abstracts a group membership API across a number of drivers. 
> I don't need that. I need a way to maintain a service record (with 
> maintenance period information, region, and an evolvable data record 
> format) and query those service records in an RDBMS-like manner but 
> without the RDBMS being involved.
> 
> >> servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
> >> service liveness checking, which should dramatically reduce the amount of 
> >> both
> >> DB traffic as well as conductor/MQ service update traffic.
> >
> > Interesting. Joshua and Vilob tried to push usage of tooz group
> > membership a couple of years ago, but it got nowhere. Well, no, they got
> > 2 specs written IIRC:
> >
> >   
> > https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
> >
> > But then it died for whatever reasons on Nova side.
> 
> It died because it didn't actually solve a problem.
> 
> The problem is that even if we incorporate tooz, we would still need to 
> have a service table in the RDBMS and continue to query it over and over 
> again in the scheduler and API nodes.
> 

Most likely it was designed with hesitance to have a tooz requirement
to be a source of truth. But it's certainly not a problem for most tooz
backends to be a source of truth. Certainly not for etcd or ZK, which
are both designed to be that.

> I want all service information in the same place, and I don't want to 
> use an RDBMS for that information. etcd3 provides an ideal place to 
> store service record information. Google Protocol Buffers is an ideal 
> data format for evolvable versioned objects. os-lively presents an API 
> that solves the problem I want to solve in Nova. tooz didn't.
> 

Was there something inherent in tooz's design that prevented you from
adding it to tooz's group API? Said API already includes liveness (watch
the group that corresponds to the service you want).

The only thing missing is being able to get groups and group members
by secondary indexes. etcd3's built in indexes by field are pretty nice
for that, but ZK can likely also do it too by maintaining the index in
the driver.

I understand abstractions can seem pretty cumbersome when you're moving
fast. It's not something I want to see stand in your way. But it would
be nice to see where there's deficiency in tooz so we can be there for
the next project that needs it and maybe eventually factor out direct
etcd3 usage so users who have maybe chosen ZK as their tooz backend can
also benefit from your work.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Jay Pipes]

On 03/14/2017 04:22 PM, Joshua Harlow wrote:

Jay Pipes wrote:

On 03/14/2017 02:50 PM, Julien Danjou wrote:

On Tue, Mar 14 2017, Jay Pipes wrote:


Not tooz, because I'm not interested in a DLM nor leader election
library
(that's what the underlying etcd3 cluster handles for me), only a
fast service
liveness/healthcheck system, but it shows usage of etcd3 and Google
Protocol
Buffers implementing a simple API for liveness checking and host
maintenance
reporting.


Cool cool. So that's the same feature that we implemented in tooz 3
years ago. It's called "group membership". You create a group, make
nodes join it, and you know who's dead/alive and get notified when their
status change.


The point of os-lively is not to provide a thin API over ZooKeeper's
group membership interface. The point of os-lively is to remove the need
to have a database (RDBMS) record of a service in Nova.

tooz simply abstracts a group membership API across a number of drivers.
I don't need that. I need a way to maintain a service record (with
maintenance period information, region, and an evolvable data record
format) and query those service records in an RDBMS-like manner but
without the RDBMS being involved.


My plan is to push some proof-of-concept patches that replace Nova's
servicegroup API with os-lively and eliminate Nova's use of an RDBMS
for
service liveness checking, which should dramatically reduce the
amount of both
DB traffic as well as conductor/MQ service update traffic.


Interesting. Joshua and Vilob tried to push usage of tooz group
membership a couple of years ago, but it got nowhere. Well, no, they got
2 specs written IIRC:

https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html

But then it died for whatever reasons on Nova side.


It died because it didn't actually solve a problem.


Hmmm, idk about that, more likely other things involved, but point taken
(and not meant personally).


That should have read "it died because it didn't actually solve *the* 
problem". Meaning, the problem of having to store service and 
maintenance information in the RDBMS. Sorry, I didn't mean that tooz 
doesn't solve problems. That's not at all how I meant to come across!



The problem is that even if we incorporate tooz, we would still need to
have a service table in the RDBMS and continue to query it over and over
again in the scheduler and API nodes.

I want all service information in the same place, and I don't want to
use an RDBMS for that information. etcd3 provides an ideal place to
store service record information. Google Protocol Buffers is an ideal
data format for evolvable versioned objects. os-lively presents an API
that solves the problem I want to solve in Nova. tooz didn't.


Def looks like u are doing some custom service indexes and such in etcd,
so ya, the default in tooz may not fit that kind of specialized model
(though I can't say such a model would be unique to nova).

https://gist.github.com/harlowja/57394357e81703a595a15d6dd7c774eb was
something I threw together, tooz may not be a perfect match, but still
seems like it can evolve to store something like your indexes @
https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L524-L542




Best,
-jay

__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Joshua Harlow]

Jay Pipes wrote:

On 03/14/2017 02:50 PM, Julien Danjou wrote:

On Tue, Mar 14 2017, Jay Pipes wrote:


Not tooz, because I'm not interested in a DLM nor leader election
library
(that's what the underlying etcd3 cluster handles for me), only a
fast service
liveness/healthcheck system, but it shows usage of etcd3 and Google
Protocol
Buffers implementing a simple API for liveness checking and host
maintenance
reporting.


Cool cool. So that's the same feature that we implemented in tooz 3
years ago. It's called "group membership". You create a group, make
nodes join it, and you know who's dead/alive and get notified when their
status change.


The point of os-lively is not to provide a thin API over ZooKeeper's
group membership interface. The point of os-lively is to remove the need
to have a database (RDBMS) record of a service in Nova.

tooz simply abstracts a group membership API across a number of drivers.
I don't need that. I need a way to maintain a service record (with
maintenance period information, region, and an evolvable data record
format) and query those service records in an RDBMS-like manner but
without the RDBMS being involved.


My plan is to push some proof-of-concept patches that replace Nova's
servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
service liveness checking, which should dramatically reduce the
amount of both
DB traffic as well as conductor/MQ service update traffic.


Interesting. Joshua and Vilob tried to push usage of tooz group
membership a couple of years ago, but it got nowhere. Well, no, they got
2 specs written IIRC:

https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html


But then it died for whatever reasons on Nova side.


It died because it didn't actually solve a problem.


Hmmm, idk about that, more likely other things involved, but point taken 
(and not meant personally).




The problem is that even if we incorporate tooz, we would still need to
have a service table in the RDBMS and continue to query it over and over
again in the scheduler and API nodes.

I want all service information in the same place, and I don't want to
use an RDBMS for that information. etcd3 provides an ideal place to
store service record information. Google Protocol Buffers is an ideal
data format for evolvable versioned objects. os-lively presents an API
that solves the problem I want to solve in Nova. tooz didn't.


Def looks like u are doing some custom service indexes and such in etcd, 
so ya, the default in tooz may not fit that kind of specialized model 
(though I can't say such a model would be unique to nova).


https://gist.github.com/harlowja/57394357e81703a595a15d6dd7c774eb was 
something I threw together, tooz may not be a perfect match, but still 
seems like it can evolve to store something like your indexes @ 
https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L524-L542 





Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Jay Pipes]

On 03/14/2017 02:50 PM, Julien Danjou wrote:

On Tue, Mar 14 2017, Jay Pipes wrote:


Not tooz, because I'm not interested in a DLM nor leader election library
(that's what the underlying etcd3 cluster handles for me), only a fast service
liveness/healthcheck system, but it shows usage of etcd3 and Google Protocol
Buffers implementing a simple API for liveness checking and host maintenance
reporting.


Cool cool. So that's the same feature that we implemented in tooz 3
years ago. It's called "group membership". You create a group, make
nodes join it, and you know who's dead/alive and get notified when their
status change.


The point of os-lively is not to provide a thin API over ZooKeeper's 
group membership interface. The point of os-lively is to remove the need 
to have a database (RDBMS) record of a service in Nova.


tooz simply abstracts a group membership API across a number of drivers. 
I don't need that. I need a way to maintain a service record (with 
maintenance period information, region, and an evolvable data record 
format) and query those service records in an RDBMS-like manner but 
without the RDBMS being involved.



My plan is to push some proof-of-concept patches that replace Nova's
servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
service liveness checking, which should dramatically reduce the amount of both
DB traffic as well as conductor/MQ service update traffic.


Interesting. Joshua and Vilob tried to push usage of tooz group
membership a couple of years ago, but it got nowhere. Well, no, they got
2 specs written IIRC:

  
https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html

But then it died for whatever reasons on Nova side.


It died because it didn't actually solve a problem.

The problem is that even if we incorporate tooz, we would still need to 
have a service table in the RDBMS and continue to query it over and over 
again in the scheduler and API nodes.


I want all service information in the same place, and I don't want to 
use an RDBMS for that information. etcd3 provides an ideal place to 
store service record information. Google Protocol Buffers is an ideal 
data format for evolvable versioned objects. os-lively presents an API 
that solves the problem I want to solve in Nova. tooz didn't.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Julien Danjou]

On Tue, Mar 14 2017, Jay Pipes wrote:

> Not tooz, because I'm not interested in a DLM nor leader election library
> (that's what the underlying etcd3 cluster handles for me), only a fast service
> liveness/healthcheck system, but it shows usage of etcd3 and Google Protocol
> Buffers implementing a simple API for liveness checking and host maintenance
> reporting.

Cool cool. So that's the same feature that we implemented in tooz 3
years ago. It's called "group membership". You create a group, make
nodes join it, and you know who's dead/alive and get notified when their
status change.

> My plan is to push some proof-of-concept patches that replace Nova's
> servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
> service liveness checking, which should dramatically reduce the amount of both
> DB traffic as well as conductor/MQ service update traffic.

Interesting. Joshua and Vilob tried to push usage of tooz group
membership a couple of years ago, but it got nowhere. Well, no, they got
2 specs written IIRC:

  
https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html

But then it died for whatever reasons on Nova side.

Cheers,
-- 
Julien Danjou
/* Free Software hacker
   https://julien.danjou.info */


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Fox, Kevin M]

+1

From: Thierry Carrez [thie...@openstack.org]
Sent: Tuesday, March 14, 2017 3:00 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for 
Tooz/DLM

Fox, Kevin M wrote:
> With my operator hat on, I would like to use the etcd backend, as I'm already 
> paying the cost of maintaining etcd clusters as part of Kubernetes. Adding 
> Zookeeper is a lot more work.

+1

In the spirit of better operationally integrating with Kubernetes, I
think we need to support etcd, at least as an option.

As I mentioned in another thread, for base services like databases,
message queues and distributed lock managers, the Architecture WG
started to promote an expand/contract model. Start by supporting a
couple viable options, and then once operators / the market decides on
one winner, contract to only supporting that winner, and start using the
specific features of that technology.

For databases and message queues, it's more than time for us to
contract. For DLMs, we are in the expand phase. We should only support a
very limited set of valuable options though -- no need to repeat the
mistakes of the past and support a dozen options just because we can.
Here it seems Zookeeper gives us the mature / featureful angle, and etcd
covers the Kubernetes cooperation / non-Java angle. I don't really see
the point of supporting a third option.

--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Jay Pipes]

On 03/14/2017 08:57 AM, Julien Danjou wrote:

On Tue, Mar 14 2017, Davanum Srinivas wrote:


Let's do it!! (etcd v2-v3 in tooz)


Hehe. I'll move that higher in my priority list, I swear. But anyone is
free to beat me to it in the meantime. ;)


A weekend experiment:

https://github.com/jaypipes/os-lively

Not tooz, because I'm not interested in a DLM nor leader election 
library (that's what the underlying etcd3 cluster handles for me), only 
a fast service liveness/healthcheck system, but it shows usage of etcd3 
and Google Protocol Buffers implementing a simple API for liveness 
checking and host maintenance reporting.


My plan is to push some proof-of-concept patches that replace Nova's 
servicegroup API with os-lively and eliminate Nova's use of an RDBMS for 
service liveness checking, which should dramatically reduce the amount 
of both DB traffic as well as conductor/MQ service update traffic.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Tomasz Pa]

Etcd seems to be a better choice for performance reasons as well:

https://coreos.com/blog/performance-of-etcd.html

TP

On 14 Mar 2017 12:45 am, "Davanum Srinivas"  wrote:

> On Tue, Mar 14 2017, Davanum Srinivas wrote:
>
> > Let's do it!! (etcd v2-v3 in tooz)
>
> Hehe. I'll move that higher in my priority list, I swear. But anyone is
> free to beat me to it in the meantime. ;)
>
> --
> Julien Danjou
> -- Free Software hacker
> -- https://julien.danjou.info
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Davanum Srinivas]

Julien,

Let's do it!! (etcd v2-v3 in tooz)

Thanks,
Dims

On Tue, Mar 14, 2017 at 5:24 AM, Julien Danjou  wrote:
> On Mon, Mar 13 2017, Joshua Harlow wrote:
>
>> Etcd I think is also switching to grpc sometime in the future (afaik); that
>> feature is in alpha/?beta?/experimental right now.
>
> Yeah I think that's the main "problem" in tooz right now, it's that we
> still rely on etcd v2 and I think v3 is out with that. And the driver
> would need to be updated.
>
> I've that on my plate for a while now, but it never got urgent now. I'm
> still willing to work on it, especially if people are interested in
> using it. We would sure be for Telemetry.
>
> --
> Julien Danjou
> ;; Free Software hacker
> ;; https://julien.danjou.info
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Thierry Carrez]

Fox, Kevin M wrote:
> With my operator hat on, I would like to use the etcd backend, as I'm already 
> paying the cost of maintaining etcd clusters as part of Kubernetes. Adding 
> Zookeeper is a lot more work.

+1

In the spirit of better operationally integrating with Kubernetes, I
think we need to support etcd, at least as an option.

As I mentioned in another thread, for base services like databases,
message queues and distributed lock managers, the Architecture WG
started to promote an expand/contract model. Start by supporting a
couple viable options, and then once operators / the market decides on
one winner, contract to only supporting that winner, and start using the
specific features of that technology.

For databases and message queues, it's more than time for us to
contract. For DLMs, we are in the expand phase. We should only support a
very limited set of valuable options though -- no need to repeat the
mistakes of the past and support a dozen options just because we can.
Here it seems Zookeeper gives us the mature / featureful angle, and etcd
covers the Kubernetes cooperation / non-Java angle. I don't really see
the point of supporting a third option.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Julien Danjou]

On Mon, Mar 13 2017, Joshua Harlow wrote:

> Etcd I think is also switching to grpc sometime in the future (afaik); that
> feature is in alpha/?beta?/experimental right now.

Yeah I think that's the main "problem" in tooz right now, it's that we
still rely on etcd v2 and I think v3 is out with that. And the driver
would need to be updated.

I've that on my plate for a while now, but it never got urgent now. I'm
still willing to work on it, especially if people are interested in
using it. We would sure be for Telemetry.

-- 
Julien Danjou
;; Free Software hacker
;; https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Joshua Harlow]

Clint Byrum wrote:

Excerpts from Fox, Kevin M's message of 2017-03-14 00:09:55 +:

With my operator hat on, I would like to use the etcd backend, as I'm already 
paying the cost of maintaining etcd clusters as part of Kubernetes. Adding 
Zookeeper is a lot more work.



It would probably be a good idea to put etcd in as a plugin and make
sure it is tested in the gate. IIRC nobody has requested the one thing
ZK can do that none of the others can (fair locking), but it's likely
there are bugs in both drivers since IIRC neither are enabled in any
gate.


Pretty sure etcd can do fair locking now :-P

Or reading 
https://coreos.com/etcd/docs/latest/v2/api.html#atomically-creating-in-order-keys 
it seems like we should be able to.


Honestly the APIs of both etcd and zookeeper seem pretty equivalent; 
etcd is a little more k/v oriented (and has key ttls, to a degree) but 
the rest seems nearly identical (which isn't a bad thing).


Etcd I think is also switching to grpc sometime in the future (afaik); 
that feature is in alpha/?beta?/experimental right now.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Clint Byrum]

Excerpts from Fox, Kevin M's message of 2017-03-14 00:09:55 +:
> With my operator hat on, I would like to use the etcd backend, as I'm already 
> paying the cost of maintaining etcd clusters as part of Kubernetes. Adding 
> Zookeeper is a lot more work.
> 

It would probably be a good idea to put etcd in as a plugin and make
sure it is tested in the gate. IIRC nobody has requested the one thing
ZK can do that none of the others can (fair locking), but it's likely
there are bugs in both drivers since IIRC neither are enabled in any
gate.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Fox, Kevin M]

With my operator hat on, I would like to use the etcd backend, as I'm already 
paying the cost of maintaining etcd clusters as part of Kubernetes. Adding 
Zookeeper is a lot more work.

Thanks,
Kevin

From: Davanum Srinivas [dava...@gmail.com]
Sent: Monday, March 13, 2017 4:42 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

Folks,

Currently devstack defaults to zookeeper and is opinionated about it.
Talking to Josh, etcd seems to be a good option too.

Does anyone have specific experience with etcd with tooz?
Do we want to make DLM mandatory? (always run with tooz + backend)?

Asking because lots of the container related projects are using etcd
[1], so we may want to avoid both zookeeper and etcd?

Thoughts please. Note that this came up during the Atlanta PTG as well [2]

Thanks,
Dims

PS: the dragonflow one looks really good! [3]

[1] http://codesearch.openstack.org/?q=etcd=nope=devstack%2F.*=
[2] https://etherpad.openstack.org/p/ptg-architecture-workgroup
[3] http://git.openstack.org/cgit/openstack/dragonflow/tree/devstack/etcd_driver
--
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Davanum Srinivas]

Folks,

Currently devstack defaults to zookeeper and is opinionated about it.
Talking to Josh, etcd seems to be a good option too.

Does anyone have specific experience with etcd with tooz?
Do we want to make DLM mandatory? (always run with tooz + backend)?

Asking because lots of the container related projects are using etcd
[1], so we may want to avoid both zookeeper and etcd?

Thoughts please. Note that this came up during the Atlanta PTG as well [2]

Thanks,
Dims

PS: the dragonflow one looks really good! [3]

[1] http://codesearch.openstack.org/?q=etcd=nope=devstack%2F.*=
[2] https://etherpad.openstack.org/p/ptg-architecture-workgroup
[3] http://git.openstack.org/cgit/openstack/dragonflow/tree/devstack/etcd_driver
-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev