Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance
For people who have been following this topic, a reminder that later today there will be a TC meeting dedicated to discussing this issues captured by this thread[1], the related thread on active or passive database approaches[2] and the two reviews about "what to do about postgreSQL" [3][4]. It will be today (6 June) at 20.00 UTC. [1] http://lists.openstack.org/pipermail/openstack-dev/2017-May/116642.html [2] http://lists.openstack.org/pipermail/openstack-dev/2017-May/117148.html [3] https://review.openstack.org/#/c/427880/ [4] https://review.openstack.org/#/c/465589/ -- Chris Dent ┬──┬◡ノ(° -°ノ) https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance
Thanks. It’s more of a question of not leaving people high and dry when they have made a reasonable choice in the past based on the choices supported at the time. Tim On 23.05.17, 21:14, "Sean Dague" wrote: On 05/23/2017 02:35 PM, Tim Bell wrote: > Is there a proposal where deployments who chose Postgres on good faith can find migration path to a MySQL based solution? Yes, a migration tool exploration is action #2 in the current proposal. Also, to be clear, we're not at the stage of removing anything at this point. We're mostly just signaling to people where the nice paved road is, and where the gravel road is. It's like the signs in the spring on the road where frost heaves are (at least in the North East US). -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance
On 05/23/2017 02:35 PM, Tim Bell wrote: > Is there a proposal where deployments who chose Postgres on good faith can > find migration path to a MySQL based solution? Yes, a migration tool exploration is action #2 in the current proposal. Also, to be clear, we're not at the stage of removing anything at this point. We're mostly just signaling to people where the nice paved road is, and where the gravel road is. It's like the signs in the spring on the road where frost heaves are (at least in the North East US). -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance
Is there a proposal where deployments who chose Postgres on good faith can find migration path to a MySQL based solution? Tim On 23.05.17, 18:35, "Octave J. Orgeron" wrote: As OpenStack has evolved and grown, we are ending up with more and more MySQL-isms in the code. I'd love to see OpenStack support every database out there, but that is becoming more and more difficult. I've tried to get OpenStack to work with other databases like Oracle DB, MongoDB, TimesTen, NoSQL, and I can tell you that first hand it's not doable without making some significant changes. Some services would be easy to make more database agnostic, but most would require a lot of reworking. I think the pragmatic thing is to do is focus on supporting the MySQL dialect with the different engines and clustering technologies that have emerged. oslo_db is a great abstraction layer. We should continue to build upon that and make sure that every OpenStack service uses it end-to-end. I've already seen plenty of cases where services like Barbican and Murano are not using it. I've also seen plenty of use cases where core services are using the older methods of connecting to the database and re-inventing the wheel to deal with things like retries. The more we use oslo_db and make sure that people are consistent with it's use and best practices, we better off we'll be in the long-run. On the topic of doing live upgrades. I think it's a "nice to have" feature, but again we need a consistent framework that all services will follow. It's already complicated enough with how different services deal with parallelism and locking. So if we are going to go down this path across even the core services, we need to have a solid solution and framework. Otherwise, we'll end up with a hodgepodge of maturity levels between services. The expectation from operators is that if you say you can do live upgrades, they will expect that to be the case across all of OpenStack and not a buffet style feature. We would also have to take into consideration larger shops that have more distributed and scaled-out control planes. So we need be careful on this as it will have a wide impact on development, testing, and operating. Octave On 5/23/2017 6:00 AM, Sean Dague wrote: > On 05/22/2017 11:26 PM, Matt Riedemann wrote: >> On 5/22/2017 10:58 AM, Sean Dague wrote: >>> I think these are actually compatible concerns. The current proposal to >>> me actually tries to address A1 & B1, with a hint about why A2 is >>> valuable and we would want to do that. >>> >>> It feels like there would be a valuable follow on in which A2 & B2 were >>> addressed which is basically "progressive enhancements can be allowed to >>> only work with MySQL based backends". Which is the bit that Monty has >>> been pushing for in other threads. >>> >>> This feels like what a Tier 2 support looks like. A basic SQLA and pray >>> so that if you live behind SQLA you are probably fine (though not >>> tested), and then test and advanced feature roll out on a single >>> platform. Any of that work might port to other platforms over time, but >>> we don't want to make that table stakes for enhancements. >> I think this is reasonable and is what I've been hoping for as a result >> of the feedback on this. >> >> I think it's totally fine to say tier 1 backends get shiny new features. >> I mean, hell, compare the libvirt driver in nova to all other virt >> drivers in nova. New features are written for the libvirt driver and we >> have to strong-arm them into other drivers for a compatibility story. >> >> I think we should turn on postgresql as a backend in one of the CI jobs, >> as I've noted in the governance change - it could be the nova-next >> non-voting job which only runs on nova, but we should have something >> testing this as long as it's around, especially given how easy it is to >> turn this on in upstream CI (it's flipping a devstack variable). > Postgresql support shouldn't be in devstack. If we're taking a tier 2 > approach, someone needs to carve out database plugins from devstack and > pg would be one (as could be galera, etc). > > This historical artifact that pg was maintained in devstack, but much > more widely used backends were not, is part of the issue. > > It would also be a good unit test case as to whether there are pg > focused folks around out there willing to do this basic devstack plugin > / job setup work. > > -Sean > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://l
Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance
As OpenStack has evolved and grown, we are ending up with more and more MySQL-isms in the code. I'd love to see OpenStack support every database out there, but that is becoming more and more difficult. I've tried to get OpenStack to work with other databases like Oracle DB, MongoDB, TimesTen, NoSQL, and I can tell you that first hand it's not doable without making some significant changes. Some services would be easy to make more database agnostic, but most would require a lot of reworking. I think the pragmatic thing is to do is focus on supporting the MySQL dialect with the different engines and clustering technologies that have emerged. oslo_db is a great abstraction layer. We should continue to build upon that and make sure that every OpenStack service uses it end-to-end. I've already seen plenty of cases where services like Barbican and Murano are not using it. I've also seen plenty of use cases where core services are using the older methods of connecting to the database and re-inventing the wheel to deal with things like retries. The more we use oslo_db and make sure that people are consistent with it's use and best practices, we better off we'll be in the long-run. On the topic of doing live upgrades. I think it's a "nice to have" feature, but again we need a consistent framework that all services will follow. It's already complicated enough with how different services deal with parallelism and locking. So if we are going to go down this path across even the core services, we need to have a solid solution and framework. Otherwise, we'll end up with a hodgepodge of maturity levels between services. The expectation from operators is that if you say you can do live upgrades, they will expect that to be the case across all of OpenStack and not a buffet style feature. We would also have to take into consideration larger shops that have more distributed and scaled-out control planes. So we need be careful on this as it will have a wide impact on development, testing, and operating. Octave On 5/23/2017 6:00 AM, Sean Dague wrote: On 05/22/2017 11:26 PM, Matt Riedemann wrote: On 5/22/2017 10:58 AM, Sean Dague wrote: I think these are actually compatible concerns. The current proposal to me actually tries to address A1 & B1, with a hint about why A2 is valuable and we would want to do that. It feels like there would be a valuable follow on in which A2 & B2 were addressed which is basically "progressive enhancements can be allowed to only work with MySQL based backends". Which is the bit that Monty has been pushing for in other threads. This feels like what a Tier 2 support looks like. A basic SQLA and pray so that if you live behind SQLA you are probably fine (though not tested), and then test and advanced feature roll out on a single platform. Any of that work might port to other platforms over time, but we don't want to make that table stakes for enhancements. I think this is reasonable and is what I've been hoping for as a result of the feedback on this. I think it's totally fine to say tier 1 backends get shiny new features. I mean, hell, compare the libvirt driver in nova to all other virt drivers in nova. New features are written for the libvirt driver and we have to strong-arm them into other drivers for a compatibility story. I think we should turn on postgresql as a backend in one of the CI jobs, as I've noted in the governance change - it could be the nova-next non-voting job which only runs on nova, but we should have something testing this as long as it's around, especially given how easy it is to turn this on in upstream CI (it's flipping a devstack variable). Postgresql support shouldn't be in devstack. If we're taking a tier 2 approach, someone needs to carve out database plugins from devstack and pg would be one (as could be galera, etc). This historical artifact that pg was maintained in devstack, but much more widely used backends were not, is part of the issue. It would also be a good unit test case as to whether there are pg focused folks around out there willing to do this basic devstack plugin / job setup work. -Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance
On 05/22/2017 11:26 PM, Matt Riedemann wrote: > On 5/22/2017 10:58 AM, Sean Dague wrote: >> I think these are actually compatible concerns. The current proposal to >> me actually tries to address A1 & B1, with a hint about why A2 is >> valuable and we would want to do that. >> >> It feels like there would be a valuable follow on in which A2 & B2 were >> addressed which is basically "progressive enhancements can be allowed to >> only work with MySQL based backends". Which is the bit that Monty has >> been pushing for in other threads. >> >> This feels like what a Tier 2 support looks like. A basic SQLA and pray >> so that if you live behind SQLA you are probably fine (though not >> tested), and then test and advanced feature roll out on a single >> platform. Any of that work might port to other platforms over time, but >> we don't want to make that table stakes for enhancements. > > I think this is reasonable and is what I've been hoping for as a result > of the feedback on this. > > I think it's totally fine to say tier 1 backends get shiny new features. > I mean, hell, compare the libvirt driver in nova to all other virt > drivers in nova. New features are written for the libvirt driver and we > have to strong-arm them into other drivers for a compatibility story. > > I think we should turn on postgresql as a backend in one of the CI jobs, > as I've noted in the governance change - it could be the nova-next > non-voting job which only runs on nova, but we should have something > testing this as long as it's around, especially given how easy it is to > turn this on in upstream CI (it's flipping a devstack variable). Postgresql support shouldn't be in devstack. If we're taking a tier 2 approach, someone needs to carve out database plugins from devstack and pg would be one (as could be galera, etc). This historical artifact that pg was maintained in devstack, but much more widely used backends were not, is part of the issue. It would also be a good unit test case as to whether there are pg focused folks around out there willing to do this basic devstack plugin / job setup work. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance
On Mon, 22 May 2017, Sean Dague wrote: This feels like what a Tier 2 support looks like. A basic SQLA and pray so that if you live behind SQLA you are probably fine (though not tested), and then test and advanced feature roll out on a single platform. Any of that work might port to other platforms over time, but we don't want to make that table stakes for enhancements. I've often wondered why what's being called "Tier 1" (advancec features) here isn't something done downstream of "generic" OpenStack. Which is not to say it would have to be closed source or vendor oriented. Simply not here. It may be we've got enough to deal with here. The 'external' model described by Monty makes things that are not here easier to manage (but, to be fair, not necessarily easier to make). -- Chris Dent ┬──┬◡ノ(° -°ノ) https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance
On 5/22/2017 10:58 AM, Sean Dague wrote: I think these are actually compatible concerns. The current proposal to me actually tries to address A1 & B1, with a hint about why A2 is valuable and we would want to do that. It feels like there would be a valuable follow on in which A2 & B2 were addressed which is basically "progressive enhancements can be allowed to only work with MySQL based backends". Which is the bit that Monty has been pushing for in other threads. This feels like what a Tier 2 support looks like. A basic SQLA and pray so that if you live behind SQLA you are probably fine (though not tested), and then test and advanced feature roll out on a single platform. Any of that work might port to other platforms over time, but we don't want to make that table stakes for enhancements. I think this is reasonable and is what I've been hoping for as a result of the feedback on this. I think it's totally fine to say tier 1 backends get shiny new features. I mean, hell, compare the libvirt driver in nova to all other virt drivers in nova. New features are written for the libvirt driver and we have to strong-arm them into other drivers for a compatibility story. I think we should turn on postgresql as a backend in one of the CI jobs, as I've noted in the governance change - it could be the nova-next non-voting job which only runs on nova, but we should have something testing this as long as it's around, especially given how easy it is to turn this on in upstream CI (it's flipping a devstack variable). -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] revised Postgresql deprecation patch for governance
On 05/15/2017 07:16 AM, Sean Dague wrote: > We had a forum session in Boston on Postgresql and out of that agreed to > the following steps forward: > > 1. explicitly warn in operator facing documentation that Postgresql is > less supported than MySQL. This was deemed better than just removing > documentation, because when people see Postgresql files in tree they'll > make assumptions (at least one set of operators did). > > 2. Suse is in process of investigating migration from PG to Gallera for > future versions of their OpenStack product. They'll make their findings > and tooling open to help determine how burdensome this kind of > transition would be for folks. > > After those findings, we can come back with any next steps (or just > leave it as good enough there). > > The TC governance patch is updated here - > https://review.openstack.org/#/c/427880/ - or if there are other > discussion questions feel free to respond to this thread. I've ended up in a number of conversations publicly and privately over the last week. I'm trying to figure out how we best capture and acknowledge the concerns. My top concerns remain: A1) Do not surprise users late by them only finding out they are on the less traveled once they are so deeply committed there is no turning back. It's fine for users to choose that path as long as they are informed that they are going to need to be more self reliant. A2) Do not prevent features like zero downtime keystone making forward progress with a MySQL only solution. There will always be a way to handle these things with a change window, but the non change window version really does need more understanding of what the db is doing. There are some orthogonal concerns B1) PG was chosen by people in the past, maybe more than we realized, that's real users that we don't want to throw under a bus. Whole sale delete is off the table. Even what deprecation might mean is hard to figure out given that there is "no clear path off", "missing data of who's on it", and potentially creative solutions using it that people would like (the Cockroach db question, though given some of the Galera fixes that have had to go in, these things are never drop in replacements). B2) The upstream code isn't so irreparably changed (e.g. delete the SQLA layer) that it's not possible to have alternative DB backends (especially as people might want to experiement with different approaches in the future). I think these are actually compatible concerns. The current proposal to me actually tries to address A1 & B1, with a hint about why A2 is valuable and we would want to do that. It feels like there would be a valuable follow on in which A2 & B2 were addressed which is basically "progressive enhancements can be allowed to only work with MySQL based backends". Which is the bit that Monty has been pushing for in other threads. This feels like what a Tier 2 support looks like. A basic SQLA and pray so that if you live behind SQLA you are probably fine (though not tested), and then test and advanced feature roll out on a single platform. Any of that work might port to other platforms over time, but we don't want to make that table stakes for enhancements. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tc] revised Postgresql deprecation patch for governance
We had a forum session in Boston on Postgresql and out of that agreed to the following steps forward: 1. explicitly warn in operator facing documentation that Postgresql is less supported than MySQL. This was deemed better than just removing documentation, because when people see Postgresql files in tree they'll make assumptions (at least one set of operators did). 2. Suse is in process of investigating migration from PG to Gallera for future versions of their OpenStack product. They'll make their findings and tooling open to help determine how burdensome this kind of transition would be for folks. After those findings, we can come back with any next steps (or just leave it as good enough there). The TC governance patch is updated here - https://review.openstack.org/#/c/427880/ - or if there are other discussion questions feel free to respond to this thread. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev