Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance

[Chris Dent]

For people who have been following this topic, a reminder that later
today there will be a TC meeting dedicated to discussing this issues
captured by this thread[1], the related thread on active or passive
database approaches[2] and the two reviews about "what to do about
postgreSQL" [3][4].

It will be today (6 June) at 20.00 UTC.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-May/116642.html
[2] http://lists.openstack.org/pipermail/openstack-dev/2017-May/117148.html
[3] https://review.openstack.org/#/c/427880/
[4] https://review.openstack.org/#/c/465589/

--
Chris Dent  ┬──┬◡ﾉ(° -°ﾉ)   https://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance

[Tim Bell]

Thanks. It’s more of a question of not leaving people high and dry when they 
have made a reasonable choice in the past based on the choices supported at the 
time.

Tim

On 23.05.17, 21:14, "Sean Dague"  wrote:

On 05/23/2017 02:35 PM, Tim Bell wrote:
> Is there a proposal where deployments who chose Postgres on good faith 
can find migration path to a MySQL based solution?

Yes, a migration tool exploration is action #2 in the current proposal.

Also, to be clear, we're not at the stage of removing anything at this
point. We're mostly just signaling to people where the nice paved road
is, and where the gravel road is. It's like the signs in the spring
 on the road where frost heaves are (at least in the North East US).

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance

[Sean Dague]

On 05/23/2017 02:35 PM, Tim Bell wrote:
> Is there a proposal where deployments who chose Postgres on good faith can 
> find migration path to a MySQL based solution?

Yes, a migration tool exploration is action #2 in the current proposal.

Also, to be clear, we're not at the stage of removing anything at this
point. We're mostly just signaling to people where the nice paved road
is, and where the gravel road is. It's like the signs in the spring
 on the road where frost heaves are (at least in the North East US).

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance

[Tim Bell]

Is there a proposal where deployments who chose Postgres on good faith can find 
migration path to a MySQL based solution?

Tim

On 23.05.17, 18:35, "Octave J. Orgeron"  wrote:

As OpenStack has evolved and grown, we are ending up with more and more 
MySQL-isms in the code. I'd love to see OpenStack support every database 
out there, but that is becoming more and more difficult. I've tried to 
get OpenStack to work with other databases like Oracle DB, MongoDB, 
TimesTen, NoSQL, and I can tell you that first hand it's not doable 
without making some significant changes. Some services would be easy to 
make more database agnostic, but most would require a lot of reworking. 
I think the pragmatic thing is to do is focus on supporting the MySQL 
dialect with the different engines and clustering technologies that have 
emerged. oslo_db is a great abstraction layer.  We should continue to 
build upon that and make sure that every OpenStack service uses it 
end-to-end. I've already seen plenty of cases where services like 
Barbican and Murano are not using it. I've also seen plenty of use cases 
where core services are using the older methods of connecting to the 
database and re-inventing the wheel to deal with things like retries. 
The more we use oslo_db and make sure that people are consistent with 
it's use and best practices, we better off we'll be in the long-run.

On the topic of doing live upgrades. I think it's a "nice to have" 
feature, but again we need a consistent framework that all services will 
follow. It's already complicated enough with how different services deal 
with parallelism and locking. So if we are going to go down this path 
across even the core services, we need to have a solid solution and 
framework. Otherwise, we'll end up with a hodgepodge of maturity levels 
between services. The expectation from operators is that if you say you 
can do live upgrades, they will expect that to be the case across all of 
OpenStack and not a buffet style feature. We would also have to take 
into consideration larger shops that have more distributed and 
scaled-out control planes. So we need be careful on this as it will have 
a wide impact on development, testing, and operating.

Octave


On 5/23/2017 6:00 AM, Sean Dague wrote:
> On 05/22/2017 11:26 PM, Matt Riedemann wrote:
>> On 5/22/2017 10:58 AM, Sean Dague wrote:
>>> I think these are actually compatible concerns. The current proposal to
>>> me actually tries to address A1 & B1, with a hint about why A2 is
>>> valuable and we would want to do that.
>>>
>>> It feels like there would be a valuable follow on in which A2 & B2 were
>>> addressed which is basically "progressive enhancements can be allowed to
>>> only work with MySQL based backends". Which is the bit that Monty has
>>> been pushing for in other threads.
>>>
>>> This feels like what a Tier 2 support looks like. A basic SQLA and pray
>>> so that if you live behind SQLA you are probably fine (though not
>>> tested), and then test and advanced feature roll out on a single
>>> platform. Any of that work might port to other platforms over time, but
>>> we don't want to make that table stakes for enhancements.
>> I think this is reasonable and is what I've been hoping for as a result
>> of the feedback on this.
>>
>> I think it's totally fine to say tier 1 backends get shiny new features.
>> I mean, hell, compare the libvirt driver in nova to all other virt
>> drivers in nova. New features are written for the libvirt driver and we
>> have to strong-arm them into other drivers for a compatibility story.
>>
>> I think we should turn on postgresql as a backend in one of the CI jobs,
>> as I've noted in the governance change - it could be the nova-next
>> non-voting job which only runs on nova, but we should have something
>> testing this as long as it's around, especially given how easy it is to
>> turn this on in upstream CI (it's flipping a devstack variable).
> Postgresql support shouldn't be in devstack. If we're taking a tier 2
> approach, someone needs to carve out database plugins from devstack and
> pg would be one (as could be galera, etc).
>
> This historical artifact that pg was maintained in devstack, but much
> more widely used backends were not, is part of the issue.
>
> It would also be a good unit test case as to whether there are pg
> focused folks around out there willing to do this basic devstack plugin
> / job setup work.
>
>   -Sean
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance

[Octave J. Orgeron]

As OpenStack has evolved and grown, we are ending up with more and more 
MySQL-isms in the code. I'd love to see OpenStack support every database 
out there, but that is becoming more and more difficult. I've tried to 
get OpenStack to work with other databases like Oracle DB, MongoDB, 
TimesTen, NoSQL, and I can tell you that first hand it's not doable 
without making some significant changes. Some services would be easy to 
make more database agnostic, but most would require a lot of reworking. 
I think the pragmatic thing is to do is focus on supporting the MySQL 
dialect with the different engines and clustering technologies that have 
emerged. oslo_db is a great abstraction layer.  We should continue to 
build upon that and make sure that every OpenStack service uses it 
end-to-end. I've already seen plenty of cases where services like 
Barbican and Murano are not using it. I've also seen plenty of use cases 
where core services are using the older methods of connecting to the 
database and re-inventing the wheel to deal with things like retries. 
The more we use oslo_db and make sure that people are consistent with 
it's use and best practices, we better off we'll be in the long-run.


On the topic of doing live upgrades. I think it's a "nice to have" 
feature, but again we need a consistent framework that all services will 
follow. It's already complicated enough with how different services deal 
with parallelism and locking. So if we are going to go down this path 
across even the core services, we need to have a solid solution and 
framework. Otherwise, we'll end up with a hodgepodge of maturity levels 
between services. The expectation from operators is that if you say you 
can do live upgrades, they will expect that to be the case across all of 
OpenStack and not a buffet style feature. We would also have to take 
into consideration larger shops that have more distributed and 
scaled-out control planes. So we need be careful on this as it will have 
a wide impact on development, testing, and operating.


Octave


On 5/23/2017 6:00 AM, Sean Dague wrote:

On 05/22/2017 11:26 PM, Matt Riedemann wrote:

On 5/22/2017 10:58 AM, Sean Dague wrote:

I think these are actually compatible concerns. The current proposal to
me actually tries to address A1 & B1, with a hint about why A2 is
valuable and we would want to do that.

It feels like there would be a valuable follow on in which A2 & B2 were
addressed which is basically "progressive enhancements can be allowed to
only work with MySQL based backends". Which is the bit that Monty has
been pushing for in other threads.

This feels like what a Tier 2 support looks like. A basic SQLA and pray
so that if you live behind SQLA you are probably fine (though not
tested), and then test and advanced feature roll out on a single
platform. Any of that work might port to other platforms over time, but
we don't want to make that table stakes for enhancements.

I think this is reasonable and is what I've been hoping for as a result
of the feedback on this.

I think it's totally fine to say tier 1 backends get shiny new features.
I mean, hell, compare the libvirt driver in nova to all other virt
drivers in nova. New features are written for the libvirt driver and we
have to strong-arm them into other drivers for a compatibility story.

I think we should turn on postgresql as a backend in one of the CI jobs,
as I've noted in the governance change - it could be the nova-next
non-voting job which only runs on nova, but we should have something
testing this as long as it's around, especially given how easy it is to
turn this on in upstream CI (it's flipping a devstack variable).

Postgresql support shouldn't be in devstack. If we're taking a tier 2
approach, someone needs to carve out database plugins from devstack and
pg would be one (as could be galera, etc).

This historical artifact that pg was maintained in devstack, but much
more widely used backends were not, is part of the issue.

It would also be a good unit test case as to whether there are pg
focused folks around out there willing to do this basic devstack plugin
/ job setup work.

-Sean



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance

[Sean Dague]

On 05/22/2017 11:26 PM, Matt Riedemann wrote:
> On 5/22/2017 10:58 AM, Sean Dague wrote:
>> I think these are actually compatible concerns. The current proposal to
>> me actually tries to address A1 & B1, with a hint about why A2 is
>> valuable and we would want to do that.
>>
>> It feels like there would be a valuable follow on in which A2 & B2 were
>> addressed which is basically "progressive enhancements can be allowed to
>> only work with MySQL based backends". Which is the bit that Monty has
>> been pushing for in other threads.
>>
>> This feels like what a Tier 2 support looks like. A basic SQLA and pray
>> so that if you live behind SQLA you are probably fine (though not
>> tested), and then test and advanced feature roll out on a single
>> platform. Any of that work might port to other platforms over time, but
>> we don't want to make that table stakes for enhancements.
> 
> I think this is reasonable and is what I've been hoping for as a result
> of the feedback on this.
> 
> I think it's totally fine to say tier 1 backends get shiny new features.
> I mean, hell, compare the libvirt driver in nova to all other virt
> drivers in nova. New features are written for the libvirt driver and we
> have to strong-arm them into other drivers for a compatibility story.
> 
> I think we should turn on postgresql as a backend in one of the CI jobs,
> as I've noted in the governance change - it could be the nova-next
> non-voting job which only runs on nova, but we should have something
> testing this as long as it's around, especially given how easy it is to
> turn this on in upstream CI (it's flipping a devstack variable).

Postgresql support shouldn't be in devstack. If we're taking a tier 2
approach, someone needs to carve out database plugins from devstack and
pg would be one (as could be galera, etc).

This historical artifact that pg was maintained in devstack, but much
more widely used backends were not, is part of the issue.

It would also be a good unit test case as to whether there are pg
focused folks around out there willing to do this basic devstack plugin
/ job setup work.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance

[Chris Dent]

On Mon, 22 May 2017, Sean Dague wrote:


This feels like what a Tier 2 support looks like. A basic SQLA and pray
so that if you live behind SQLA you are probably fine (though not
tested), and then test and advanced feature roll out on a single
platform. Any of that work might port to other platforms over time, but
we don't want to make that table stakes for enhancements.


I've often wondered why what's being called "Tier 1" (advancec
features) here isn't something done downstream of "generic"
OpenStack.

Which is not to say it would have to be closed source or vendor
oriented. Simply not here. It may be we've got enough to deal with
here.

The 'external' model described by Monty makes things that are not
here easier to manage (but, to be fair, not necessarily easier to
make).

--
Chris Dent  ┬──┬◡ﾉ(° -°ﾉ)   https://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance

[Matt Riedemann]

On 5/22/2017 10:58 AM, Sean Dague wrote:

I think these are actually compatible concerns. The current proposal to
me actually tries to address A1 & B1, with a hint about why A2 is
valuable and we would want to do that.

It feels like there would be a valuable follow on in which A2 & B2 were
addressed which is basically "progressive enhancements can be allowed to
only work with MySQL based backends". Which is the bit that Monty has
been pushing for in other threads.

This feels like what a Tier 2 support looks like. A basic SQLA and pray
so that if you live behind SQLA you are probably fine (though not
tested), and then test and advanced feature roll out on a single
platform. Any of that work might port to other platforms over time, but
we don't want to make that table stakes for enhancements.


I think this is reasonable and is what I've been hoping for as a result 
of the feedback on this.


I think it's totally fine to say tier 1 backends get shiny new features. 
I mean, hell, compare the libvirt driver in nova to all other virt 
drivers in nova. New features are written for the libvirt driver and we 
have to strong-arm them into other drivers for a compatibility story.


I think we should turn on postgresql as a backend in one of the CI jobs, 
as I've noted in the governance change - it could be the nova-next 
non-voting job which only runs on nova, but we should have something 
testing this as long as it's around, especially given how easy it is to 
turn this on in upstream CI (it's flipping a devstack variable).


--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance

[Sean Dague]

On 05/15/2017 07:16 AM, Sean Dague wrote:
> We had a forum session in Boston on Postgresql and out of that agreed to
> the following steps forward:
> 
> 1. explicitly warn in operator facing documentation that Postgresql is
> less supported than MySQL. This was deemed better than just removing
> documentation, because when people see Postgresql files in tree they'll
> make assumptions (at least one set of operators did).
> 
> 2. Suse is in process of investigating migration from PG to Gallera for
> future versions of their OpenStack product. They'll make their findings
> and tooling open to help determine how burdensome this kind of
> transition would be for folks.
> 
> After those findings, we can come back with any next steps (or just
> leave it as good enough there).
> 
> The TC governance patch is updated here -
> https://review.openstack.org/#/c/427880/ - or if there are other
> discussion questions feel free to respond to this thread.

I've ended up in a number of conversations publicly and privately over
the last week. I'm trying to figure out how we best capture and
acknowledge the concerns.

My top concerns remain:

A1) Do not surprise users late by them only finding out they are on the
less traveled once they are so deeply committed there is no turning
back. It's fine for users to choose that path as long as they are
informed that they are going to need to be more self reliant.

A2) Do not prevent features like zero downtime keystone making forward
progress with a MySQL only solution. There will always be a way to
handle these things with a change window, but the non change window
version really does need more understanding of what the db is doing.

There are some orthogonal concerns

B1) PG was chosen by people in the past, maybe more than we realized,
that's real users that we don't want to throw under a bus. Whole sale
delete is off the table. Even what deprecation might mean is hard to
figure out given that there is "no clear path off", "missing data of
who's on it", and potentially creative solutions using it that people
would like (the Cockroach db question, though given some of the Galera
fixes that have had to go in, these things are never drop in replacements).

B2) The upstream code isn't so irreparably changed (e.g. delete the SQLA
layer) that it's not possible to have alternative DB backends
(especially as people might want to experiement with different
approaches in the future).


I think these are actually compatible concerns. The current proposal to
me actually tries to address A1 & B1, with a hint about why A2 is
valuable and we would want to do that.

It feels like there would be a valuable follow on in which A2 & B2 were
addressed which is basically "progressive enhancements can be allowed to
only work with MySQL based backends". Which is the bit that Monty has
been pushing for in other threads.

This feels like what a Tier 2 support looks like. A basic SQLA and pray
so that if you live behind SQLA you are probably fine (though not
tested), and then test and advanced feature roll out on a single
platform. Any of that work might port to other platforms over time, but
we don't want to make that table stakes for enhancements.

-Sean


-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev