Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Chris Friesen]

On 02/07/2017 07:09 AM, Rui Chen wrote:

Actually, some users used postgresql in production deployment(8%), the following
photo extract from user survey report of April 2016.


Technically the 8% includes both production (4%) as well as dev/QA (3%), and 
proof-of-concept (1%).


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Matthew Booth]

On Thu, Feb 2, 2017 at 4:42 PM, Sean Dague  wrote:

> On 02/02/2017 10:33 AM, Mike Bayer wrote:
> >
> >
> > On 02/01/2017 10:22 AM, Monty Taylor wrote:
> >>
> >> I personally continue to be of the opinion that without an explicit
> >> vocal and well-staffed champion, supporting postgres is more trouble
> >> than it is worth. The vast majority of OpenStack deployments are on
> >> MySQL - and what's more, the code is written with MySQL in mind.
> >> Postgres and MySQL have different trade offs, different things each are
> >> good at and different places in which each has weakness. By attempting
> >> to support Postgres AND MySQL, we prevent ourselves from focusing
> >> adequate attention on making sure that our support for one of them is
> >> top-notch and in keeping with best practices for that database.
> >>
> >> So let me state my opinion slightly differently. I think we should
> >> support one and only one RDBMS backend for OpenStack, and we should open
> >> ourselves up to use advanced techniques for that backend. I don't
> >> actually care whether that DB is MySQL or Postgres - but the corpus of
> >> existing deployments on MySQL and the existing gate jobs I think make
> >> the choice one way or the other simple.
> >
> >
> > well, let me blow your mind and agree, but noting that this means, *we
> > drop SQLite also*.   IMO every openstack developer should have
> > MySQL/MariaDB running on their machine and that is part of what runs if
> > you expect to run database-related unit tests.   Targeting just one
> > database is very handy but if you really want to use the features
> > without roadblocks, you need to go all the way.
>
> That's all fine and good, we just need to rewrite about 100,000 unit
> tests to do that. I'm totally cool with someone taking that task on, but
> making a decision about postgresql shouldn't be filibustered on
> rewriting all the unit tests in OpenStack because of the ways we use
> sqlite.
>

I wrote a patch series to optionally run all our unit tests using MySQL
instead of sqlite a couple of years ago, and it wasn't that hard at the
time. The biggest issue I recall was fixing up tests which assumed
sub-second timestamp granularity which MySQL did not support at the time
(but may now).

IIRC the series died because we killed the fixture I was using in oslo.db
without replacement before my series finished landing. Fundamentally wasn't
that hard, though.

Matt
-- 
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Mike Bayer]

On 02/02/2017 11:42 AM, Sean Dague wrote:


That's all fine and good, we just need to rewrite about 100,000 unit
tests to do that. I'm totally cool with someone taking that task on, but
making a decision about postgresql shouldn't be filibustered on
rewriting all the unit tests in OpenStack because of the ways we use sqlite.


two points:

first is, you don't need to rewrite any tests, just reorganize the 
fixtures.   This is all done top-level and I've always been trying to 
get people to standardize on oslo-db built-in fixtures more anyway, this 
would ultimately make that all easier.   We would need to use an 
efficient process for tearing down of data within a schema so that tests 
can run quickly without schema rebuilds, this is all under the realm of 
"roll back the transaction" testing which oslo.db supports though nobody 
is using this right now.   It would be a big change in how things run 
and there'd be individual issues to fix but it's not a rewrite of actual 
tests.   I am not in any hurry to do any of this.


second is, I'm not a "filibuster" vote at all :).   I'm like the least 
important person in the decision chain here and I didn't even -1 the 
proposal.Deprecating Postgresql alone and doing nothing else is 
definitely very easy and would make development simpler, whereas getting 
rid of SQLite would be a much bigger job.  I'm just pointing out that we 
shouldn't pretend we "target only one database" until we get rid of 
SQLite in our test suites.



OK, third bonus point.   If we do drop postgresql support, to the degree 
that we really remove it totally from test fixtures, oslo.db 
architectures, all of that, the codebase would probably become 
mysql-specific in subtle and not-so-subtle ways pretty quickly, and 
within a few cycles we should consider that we probably will never be 
able to target multiple databases again without a much larger 
"unwinding" effort.   So while not worrying about Postgresql is handy, I 
would miss the fact that targeting two real DBs keeps us honest in terms 
of being able to target multiple databases at all, because this is a 
door that once you close we're not going to be able to open again. I 
doubt that in oslo.db itself we would realistically ever drop the 
architectures that support multiple databases, though, and as oslo.db is 
a pretty simple library it should likely continue to target postgresql 
as nothing more than a "keeping things honest" sanity check.










-Sean



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Sean Dague]

On 02/02/2017 11:48 AM, Julien Danjou wrote:
> On Wed, Feb 01 2017, Julien Danjou wrote:
> 
>> My questions are simple and can be organized in a tree:
>>
>>Does Nova want to support PostgreSQL?
>>  /  \
>>Yes  No
>>/  \
>>Why is there no gate? Ok, we'll have to remove the 
>>  PostgreSQL job in Ceilometer
> 
> Funnily enough, I don't think anyone has been able to clearly reply to
> this tree. My understanding is that Nova does not want to support
> PostgreSQL, therefore we'll remove the PostgreSQL job from Ceilometer.
> 
> The Ceilometer legacy storage engine being deprecated in Ocata, that has
> little important for us anyway.

I feel like the Answer was "No", with a pushback of "well Nova can't
make that decision in a vacuum", so we then escalated to "Ok, here is
the non vacuum solution".

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Julien Danjou]

On Wed, Feb 01 2017, Julien Danjou wrote:

> My questions are simple and can be organized in a tree:
>
>Does Nova want to support PostgreSQL?
>  /  \
>Yes  No
>/  \
>Why is there no gate? Ok, we'll have to remove the 
>  PostgreSQL job in Ceilometer

Funnily enough, I don't think anyone has been able to clearly reply to
this tree. My understanding is that Nova does not want to support
PostgreSQL, therefore we'll remove the PostgreSQL job from Ceilometer.

The Ceilometer legacy storage engine being deprecated in Ocata, that has
little important for us anyway.

Cheers,
-- 
Julien Danjou
/* Free Software hacker
   https://julien.danjou.info */


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Sean Dague]

On 02/02/2017 10:33 AM, Mike Bayer wrote:
> 
> 
> On 02/01/2017 10:22 AM, Monty Taylor wrote:
>>
>> I personally continue to be of the opinion that without an explicit
>> vocal and well-staffed champion, supporting postgres is more trouble
>> than it is worth. The vast majority of OpenStack deployments are on
>> MySQL - and what's more, the code is written with MySQL in mind.
>> Postgres and MySQL have different trade offs, different things each are
>> good at and different places in which each has weakness. By attempting
>> to support Postgres AND MySQL, we prevent ourselves from focusing
>> adequate attention on making sure that our support for one of them is
>> top-notch and in keeping with best practices for that database.
>>
>> So let me state my opinion slightly differently. I think we should
>> support one and only one RDBMS backend for OpenStack, and we should open
>> ourselves up to use advanced techniques for that backend. I don't
>> actually care whether that DB is MySQL or Postgres - but the corpus of
>> existing deployments on MySQL and the existing gate jobs I think make
>> the choice one way or the other simple.
> 
> 
> well, let me blow your mind and agree, but noting that this means, *we
> drop SQLite also*.   IMO every openstack developer should have
> MySQL/MariaDB running on their machine and that is part of what runs if
> you expect to run database-related unit tests.   Targeting just one
> database is very handy but if you really want to use the features
> without roadblocks, you need to go all the way.

That's all fine and good, we just need to rewrite about 100,000 unit
tests to do that. I'm totally cool with someone taking that task on, but
making a decision about postgresql shouldn't be filibustered on
rewriting all the unit tests in OpenStack because of the ways we use sqlite.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Jay Pipes]

On 02/02/2017 10:59 AM, Monty Taylor wrote:

On 02/02/2017 09:33 AM, Mike Bayer wrote:



On 02/01/2017 10:22 AM, Monty Taylor wrote:


I personally continue to be of the opinion that without an explicit
vocal and well-staffed champion, supporting postgres is more trouble
than it is worth. The vast majority of OpenStack deployments are on
MySQL - and what's more, the code is written with MySQL in mind.
Postgres and MySQL have different trade offs, different things each are
good at and different places in which each has weakness. By attempting
to support Postgres AND MySQL, we prevent ourselves from focusing
adequate attention on making sure that our support for one of them is
top-notch and in keeping with best practices for that database.

So let me state my opinion slightly differently. I think we should
support one and only one RDBMS backend for OpenStack, and we should open
ourselves up to use advanced techniques for that backend. I don't
actually care whether that DB is MySQL or Postgres - but the corpus of
existing deployments on MySQL and the existing gate jobs I think make
the choice one way or the other simple.



well, let me blow your mind and agree, but noting that this means, *we
drop SQLite also*.   IMO every openstack developer should have
MySQL/MariaDB running on their machine and that is part of what runs if
you expect to run database-related unit tests.   Targeting just one
database is very handy but if you really want to use the features
without roadblocks, you need to go all the way.


I could not possibly agree more strongly. Support for sqlite - which
literally nobody should EVER use in production causes much unnecessary
complexity.


I presume you don't mean for Swift.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Monty Taylor]

On 02/02/2017 09:33 AM, Mike Bayer wrote:
> 
> 
> On 02/01/2017 10:22 AM, Monty Taylor wrote:
>>
>> I personally continue to be of the opinion that without an explicit
>> vocal and well-staffed champion, supporting postgres is more trouble
>> than it is worth. The vast majority of OpenStack deployments are on
>> MySQL - and what's more, the code is written with MySQL in mind.
>> Postgres and MySQL have different trade offs, different things each are
>> good at and different places in which each has weakness. By attempting
>> to support Postgres AND MySQL, we prevent ourselves from focusing
>> adequate attention on making sure that our support for one of them is
>> top-notch and in keeping with best practices for that database.
>>
>> So let me state my opinion slightly differently. I think we should
>> support one and only one RDBMS backend for OpenStack, and we should open
>> ourselves up to use advanced techniques for that backend. I don't
>> actually care whether that DB is MySQL or Postgres - but the corpus of
>> existing deployments on MySQL and the existing gate jobs I think make
>> the choice one way or the other simple.
> 
> 
> well, let me blow your mind and agree, but noting that this means, *we
> drop SQLite also*.   IMO every openstack developer should have
> MySQL/MariaDB running on their machine and that is part of what runs if
> you expect to run database-related unit tests.   Targeting just one
> database is very handy but if you really want to use the features
> without roadblocks, you need to go all the way.

I could not possibly agree more strongly. Support for sqlite - which
literally nobody should EVER use in production causes much unnecessary
complexity.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Steve Martinelli]

On Thu, Feb 2, 2017 at 10:33 AM, Mike Bayer  wrote:
>
>
> well, let me blow your mind and agree, but noting that this means, *we
> drop SQLite also*.   IMO every openstack developer should have
> MySQL/MariaDB running on their machine and that is part of what runs if you
> expect to run database-related unit tests.   Targeting just one database is
> very handy but if you really want to use the features without roadblocks,
> you need to go all the way.


+ let's drop SQLite while we're at it
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Mike Bayer]

On 02/01/2017 10:22 AM, Monty Taylor wrote:


I personally continue to be of the opinion that without an explicit
vocal and well-staffed champion, supporting postgres is more trouble
than it is worth. The vast majority of OpenStack deployments are on
MySQL - and what's more, the code is written with MySQL in mind.
Postgres and MySQL have different trade offs, different things each are
good at and different places in which each has weakness. By attempting
to support Postgres AND MySQL, we prevent ourselves from focusing
adequate attention on making sure that our support for one of them is
top-notch and in keeping with best practices for that database.

So let me state my opinion slightly differently. I think we should
support one and only one RDBMS backend for OpenStack, and we should open
ourselves up to use advanced techniques for that backend. I don't
actually care whether that DB is MySQL or Postgres - but the corpus of
existing deployments on MySQL and the existing gate jobs I think make
the choice one way or the other simple.



well, let me blow your mind and agree, but noting that this means, *we 
drop SQLite also*.   IMO every openstack developer should have 
MySQL/MariaDB running on their machine and that is part of what runs if 
you expect to run database-related unit tests.   Targeting just one 
database is very handy but if you really want to use the features 
without roadblocks, you need to go all the way.









I agree that ceilometer should not be providing Postgres testing for the
rest of OpenStack.

Monty


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Sean Dague]

On 02/01/2017 12:24 PM, gordon chung wrote:
> 
> 
> On 01/02/17 12:15 PM, Sean Dague wrote:
>> What were you thinking about the messaging? TC resolution for
>> deprecation of postgresql as a first class backend?
>>
>> If setup tools want to support things, that's fine, just they do need to
>> realize they are owning that support its not coming from upstream.
> 
> possibly start there if we want to proceed with this or maybe it's not a 
> concern.

https://review.openstack.org/#/c/427880/ - resolution posted.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[gordon chung]

On 01/02/17 12:15 PM, Sean Dague wrote:
> What were you thinking about the messaging? TC resolution for
> deprecation of postgresql as a first class backend?
>
> If setup tools want to support things, that's fine, just they do need to
> realize they are owning that support its not coming from upstream.

possibly start there if we want to proceed with this or maybe it's not a 
concern.

tbh, i'd favour something inline with what mriedem proposed as it seems 
to be the easier path but this is based solely on supporting both 
backends in telemetry projects so maybe it's more difficult for other 
projects and they have backend-specific performance tweaks. i'm working 
under impression there is more to this than just 'drop all postgresql 
gates'.

cheers,
-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Sean Dague]

On 02/01/2017 12:21 PM, Julien Danjou wrote:
> On Wed, Feb 01 2017, Sean Dague wrote:
> 
>> If setup tools want to support things, that's fine, just they do need to
>> realize they are owning that support its not coming from upstream.
> 
> The problem as I see it right now is that upstream is schizophrenic: it
> seems it does not want to support nor gate on PostgreSQL but (have to)
> fix bug when they arise.
> 
> So what's the call?

I would call it class B support. Upstream(s) can decide to take user
fixes, we're just not going to really test them or consider it supported.

Honestly, it's basically the contract that you get with most software
that doesn't have the high level of CI you get from OpenStack. In a
finite world, you have to pick what gets the class A treatment
(including gating). The rest of it is accept user fixes if there is time.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Julien Danjou]

On Wed, Feb 01 2017, Sean Dague wrote:

> If setup tools want to support things, that's fine, just they do need to
> realize they are owning that support its not coming from upstream.

The problem as I see it right now is that upstream is schizophrenic: it
seems it does not want to support nor gate on PostgreSQL but (have to)
fix bug when they arise.

So what's the call?

-- 
Julien Danjou
// Free Software hacker
// https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Sean Dague]

On 02/01/2017 12:12 PM, gordon chung wrote:
> 
> 
> On 01/02/17 11:42 AM, Monty Taylor wrote:
>> As I mentioned before, I don't think it matters which of the two we pick
>> - although I know _way_ more about MySQL personally and it has a much
>> more proven track record at absurdly large scale - I just argue that we
>> should pick one and then actually design our database interactions based
>> on how that one works.
>>
>> I don't actually expect us to decide to do that, of course. Why would we
>> ever say no to more places for people to express their desire for
>> difference? But I strongly believe it would be in the best interests of
>> OpenStack if we did.
> 
> all for a slimmer openstack.
> 
> so Julien's initial query regarding postgresql. if we only do test one 
> and only one, i don't think it's safe to say we support anything other 
> than that and we should probably be more consistent with this messaging? 
> right now postgresql is not just gated on in places, but can just as 
> easily be configured in dev environments, and is mentioned as supported 
> in our docs[1]. we'd probably have consider all the existing deployments 
> that leverage postgresql and what response to give them on either 
> migrating or some other alternative.
> 
> [1] 
> http://docs.openstack.org/newton/install-guide-rdo/environment-sql-database.html

What were you thinking about the messaging? TC resolution for
deprecation of postgresql as a first class backend?

If setup tools want to support things, that's fine, just they do need to
realize they are owning that support its not coming from upstream.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[gordon chung]

On 01/02/17 11:42 AM, Monty Taylor wrote:
> As I mentioned before, I don't think it matters which of the two we pick
> - although I know _way_ more about MySQL personally and it has a much
> more proven track record at absurdly large scale - I just argue that we
> should pick one and then actually design our database interactions based
> on how that one works.
>
> I don't actually expect us to decide to do that, of course. Why would we
> ever say no to more places for people to express their desire for
> difference? But I strongly believe it would be in the best interests of
> OpenStack if we did.

all for a slimmer openstack.

so Julien's initial query regarding postgresql. if we only do test one 
and only one, i don't think it's safe to say we support anything other 
than that and we should probably be more consistent with this messaging? 
right now postgresql is not just gated on in places, but can just as 
easily be configured in dev environments, and is mentioned as supported 
in our docs[1]. we'd probably have consider all the existing deployments 
that leverage postgresql and what response to give them on either 
migrating or some other alternative.

[1] 
http://docs.openstack.org/newton/install-guide-rdo/environment-sql-database.html

-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Monty Taylor]

On 02/01/2017 10:06 AM, gordon chung wrote:
> 
> 
> On 01/02/17 10:22 AM, Monty Taylor wrote:
>>
>> I personally continue to be of the opinion that without an explicit
>> vocal and well-staffed champion, supporting postgres is more trouble
>> than it is worth. The vast majority of OpenStack deployments are on
>> MySQL - and what's more, the code is written with MySQL in mind.
>> Postgres and MySQL have different trade offs, different things each are
>> good at and different places in which each has weakness. By attempting
>> to support Postgres AND MySQL, we prevent ourselves from focusing
>> adequate attention on making sure that our support for one of them is
>> top-notch and in keeping with best practices for that database.
> 
> is there actually mysql only code or do you mean that once things pass 
> on gate with mysql, devs are happy? if the former, i understand not 
> wanting to have to hack postgresql to pass gate.
> 
>>
>> So let me state my opinion slightly differently. I think we should
>> support one and only one RDBMS backend for OpenStack, and we should open
>> ourselves up to use advanced techniques for that backend. I don't
>> actually care whether that DB is MySQL or Postgres - but the corpus of
>> existing deployments on MySQL and the existing gate jobs I think make
>> the choice one way or the other simple.
>>
> 
> i just want to point out that the current bug passes in mysql because 
> it's more lax in how it parses sql and allows for some 
> non-deterministism. the postgresql gate arguably caught bad sql which is 
> a good thing? ideally, i'd like to have postgresql tested as well but it 
> may be because most/all telemetry's sql backend hacks were required 
> because of mysql and not postgres.

MySQL has the ability to be stricter in what how it parses and what it
accepts. If that is a thing we value, we could totally do some work to
bump up the MySQL strictness mode and keep it there.

That said - 'bad sql' isn't a construct that exists in a vacuum. SQL is
merely a tool to talk to an engine. So while MySQL may allow some SQL
that Postgres would refuse to parse, the ultimate goal is getting data
in and out of the backend database in a manner that is efficient. That's
important (and the real thing I'm on about with all of this) because
MySQL and Postgres have very different query optimizers and deal with
indexing differently. So a query that's written "well" for one of them
can be terrible in the other and vice veras. That doesn't make either
query "invalid" - it makes that query a bad query for that engine.

For instance, InnoDB with MySQL stores rows in a B+ tree clustered by
primary key. Secondary indexes store references to the primary key,
because the clustered primary key nature makes the primary key index a
covering index so a secondary index can get to the data with two
operations. However, since the secondary index holds the primary key, in
many cases it can be much more efficient to add an artificial autoinc
primary key to the table than to use a natural primary key like a UUID,
otherwise the seconary index is going to take much more memory and scans
will have to swap in and out of buffers more. That's a table design idea
that comes from knowing how the underlying storage engine works and also
how index scans use buffers and caches during query processing. The same
design may not be a good choice in Postgres.

While it's quite awesome that SQL is somewhat of a standard-ish (not
that anyone actually cares about the actual standard, which MySQL
implements, as opposed to what is considered the de-facto standard which
is Oracle's fork that Postgres implements) - the underlying engine
differences make writing code that is portable across database engines
not worth the effort except in small or trivial applications, IMO. For
things that aim for actual massive scale, designing the database to take
advantage of the actual characteristics of the database engine is paramount.

As I mentioned before, I don't think it matters which of the two we pick
- although I know _way_ more about MySQL personally and it has a much
more proven track record at absurdly large scale - I just argue that we
should pick one and then actually design our database interactions based
on how that one works.

I don't actually expect us to decide to do that, of course. Why would we
ever say no to more places for people to express their desire for
difference? But I strongly believe it would be in the best interests of
OpenStack if we did.

> full disclaimer: i use postgres for local dev and mysql for my 
> 'production' (because packstack didn't do postgres). aka i use whatever 
> is easier to install.
> 
> cheers,
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[gordon chung]

On 01/02/17 10:22 AM, Monty Taylor wrote:
>
> I personally continue to be of the opinion that without an explicit
> vocal and well-staffed champion, supporting postgres is more trouble
> than it is worth. The vast majority of OpenStack deployments are on
> MySQL - and what's more, the code is written with MySQL in mind.
> Postgres and MySQL have different trade offs, different things each are
> good at and different places in which each has weakness. By attempting
> to support Postgres AND MySQL, we prevent ourselves from focusing
> adequate attention on making sure that our support for one of them is
> top-notch and in keeping with best practices for that database.

is there actually mysql only code or do you mean that once things pass 
on gate with mysql, devs are happy? if the former, i understand not 
wanting to have to hack postgresql to pass gate.

>
> So let me state my opinion slightly differently. I think we should
> support one and only one RDBMS backend for OpenStack, and we should open
> ourselves up to use advanced techniques for that backend. I don't
> actually care whether that DB is MySQL or Postgres - but the corpus of
> existing deployments on MySQL and the existing gate jobs I think make
> the choice one way or the other simple.
>

i just want to point out that the current bug passes in mysql because 
it's more lax in how it parses sql and allows for some 
non-deterministism. the postgresql gate arguably caught bad sql which is 
a good thing? ideally, i'd like to have postgresql tested as well but it 
may be because most/all telemetry's sql backend hacks were required 
because of mysql and not postgres.

full disclaimer: i use postgres for local dev and mysql for my 
'production' (because packstack didn't do postgres). aka i use whatever 
is easier to install.

cheers,
-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Chris Friesen]

On 02/01/2017 09:18 AM, Sean Dague wrote:

On 02/01/2017 10:12 AM, Matt Riedemann wrote:

On 2/1/2017 8:07 AM, Julien Danjou wrote:
It's not just Nova, it's the entire set of integrated-gate jobs which
dropped the postgresql job. There is a bit more history there, it's in
the ML somewhere, but it's not just like Nova decided to stop running it
in the integrated gate.

At the time it was removed, I said I thought it might be a good idea to
run it on oslo.db changes, or at least in a periodic job queue, but
since we removed it from the integrated-gate set of jobs I wouldn't just
make it gating by itself on nova again.

If it could be worked into an existing job as a wrinkle then that's an
option, i.e. nova has this placement job which was unique in newton
because it ran the optional things at the time for nova like placement
and cells v2, but those are required globally now in master so are less
special. But in Pike we're going to be looking at some new snowflakes
for that job, like cinder v3, service users, etc etc, so maybe postgres
could be worked in there.


The user survey showed almost no one actually using postgresql in
production. I think that it's time to really retire CI on postgresql
entirely. Ad hoc fixes are fine, but all the documentation and basically
all the users are on mysql, and that should be the focus moving forward.


As someone using PostgreSQL in production, that would be unfortunate. :)

Chris


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Monty Taylor]

On 02/01/2017 09:12 AM, Matt Riedemann wrote:
> On 2/1/2017 8:07 AM, Julien Danjou wrote:
>> Hi there,
>>
>> So the Ceilometer gate has been broken again by Nova today because of a
>> SQL bug in the placement API with PostgreSQL. This is the second time
>> that something like that happened: last November¹, we also reported a
>> bug in Nova wrt PostgreSQL.
>>
>> Fortunately, Mehdi investigated and sent a fix:
>>
>>   https://review.openstack.org/#/c/427667/
>>
>> So the issue is going to be resolved soon. I had hope this would not
>> happen again, but it seems Nova does not care about PostgreSQL since
>> August².
>>
>> I think I can understand that Nova does not want to support PostgreSQL
>> and therefore does not use it in the gate, however, I don't think it's a
>> rich idea to have other projects (e.g. Ceilometer) doing the gate check
>> for Nova.
>>
>> My questions are simple and can be organized in a tree:
>>
>>Does Nova want to support PostgreSQL?
>>  /  \
>>Yes  No
>>/  \
>>Why is there no gate? Ok, we'll have to remove the
>>  PostgreSQL job in Ceilometer
>>
>>
>> Thanks!
>>
>>
>> ¹ 
>> http://lists.openstack.org/pipermail/openstack-dev/2016-November/107907.html
>>
>>
>> ² 
>> http://lists.openstack.org/pipermail/openstack-dev/2016-August/101892.html
>>
>>
>>
>>
>> __
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> It's not just Nova, it's the entire set of integrated-gate jobs which
> dropped the postgresql job. There is a bit more history there, it's in
> the ML somewhere, but it's not just like Nova decided to stop running it
> in the integrated gate.

Yah. Postgres support has historically been one of those things that
occasionally someone says "we should support postgres" - and then
consistently it breaks and nobody cares enough to fix it.

> At the time it was removed, I said I thought it might be a good idea to
> run it on oslo.db changes, or at least in a periodic job queue, but
> since we removed it from the integrated-gate set of jobs I wouldn't just
> make it gating by itself on nova again.
> 
> If it could be worked into an existing job as a wrinkle then that's an
> option, i.e. nova has this placement job which was unique in newton
> because it ran the optional things at the time for nova like placement
> and cells v2, but those are required globally now in master so are less
> special. But in Pike we're going to be looking at some new snowflakes
> for that job, like cinder v3, service users, etc etc, so maybe postgres
> could be worked in there.

I personally continue to be of the opinion that without an explicit
vocal and well-staffed champion, supporting postgres is more trouble
than it is worth. The vast majority of OpenStack deployments are on
MySQL - and what's more, the code is written with MySQL in mind.
Postgres and MySQL have different trade offs, different things each are
good at and different places in which each has weakness. By attempting
to support Postgres AND MySQL, we prevent ourselves from focusing
adequate attention on making sure that our support for one of them is
top-notch and in keeping with best practices for that database.

So let me state my opinion slightly differently. I think we should
support one and only one RDBMS backend for OpenStack, and we should open
ourselves up to use advanced techniques for that backend. I don't
actually care whether that DB is MySQL or Postgres - but the corpus of
existing deployments on MySQL and the existing gate jobs I think make
the choice one way or the other simple.

I agree that ceilometer should not be providing Postgres testing for the
rest of OpenStack.

Monty


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Sean Dague]

On 02/01/2017 10:12 AM, Matt Riedemann wrote:
> On 2/1/2017 8:07 AM, Julien Danjou wrote:
> It's not just Nova, it's the entire set of integrated-gate jobs which
> dropped the postgresql job. There is a bit more history there, it's in
> the ML somewhere, but it's not just like Nova decided to stop running it
> in the integrated gate.
> 
> At the time it was removed, I said I thought it might be a good idea to
> run it on oslo.db changes, or at least in a periodic job queue, but
> since we removed it from the integrated-gate set of jobs I wouldn't just
> make it gating by itself on nova again.
> 
> If it could be worked into an existing job as a wrinkle then that's an
> option, i.e. nova has this placement job which was unique in newton
> because it ran the optional things at the time for nova like placement
> and cells v2, but those are required globally now in master so are less
> special. But in Pike we're going to be looking at some new snowflakes
> for that job, like cinder v3, service users, etc etc, so maybe postgres
> could be worked in there.

The user survey showed almost no one actually using postgresql in
production. I think that it's time to really retire CI on postgresql
entirely. Ad hoc fixes are fine, but all the documentation and basically
all the users are on mysql, and that should be the focus moving forward.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Matt Riedemann]

On 2/1/2017 8:07 AM, Julien Danjou wrote:

Hi there,

So the Ceilometer gate has been broken again by Nova today because of a
SQL bug in the placement API with PostgreSQL. This is the second time
that something like that happened: last November¹, we also reported a
bug in Nova wrt PostgreSQL.

Fortunately, Mehdi investigated and sent a fix:

  https://review.openstack.org/#/c/427667/

So the issue is going to be resolved soon. I had hope this would not
happen again, but it seems Nova does not care about PostgreSQL since
August².

I think I can understand that Nova does not want to support PostgreSQL
and therefore does not use it in the gate, however, I don't think it's a
rich idea to have other projects (e.g. Ceilometer) doing the gate check
for Nova.

My questions are simple and can be organized in a tree:

   Does Nova want to support PostgreSQL?
 /  \
   Yes  No
   /  \
   Why is there no gate? Ok, we'll have to remove the
 PostgreSQL job in Ceilometer


Thanks!


¹  http://lists.openstack.org/pipermail/openstack-dev/2016-November/107907.html

²  http://lists.openstack.org/pipermail/openstack-dev/2016-August/101892.html



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



It's not just Nova, it's the entire set of integrated-gate jobs which 
dropped the postgresql job. There is a bit more history there, it's in 
the ML somewhere, but it's not just like Nova decided to stop running it 
in the integrated gate.


At the time it was removed, I said I thought it might be a good idea to 
run it on oslo.db changes, or at least in a periodic job queue, but 
since we removed it from the integrated-gate set of jobs I wouldn't just 
make it gating by itself on nova again.


If it could be worked into an existing job as a wrinkle then that's an 
option, i.e. nova has this placement job which was unique in newton 
because it ran the optional things at the time for nova like placement 
and cells v2, but those are required globally now in master so are less 
special. But in Pike we're going to be looking at some new snowflakes 
for that job, like cinder v3, service users, etc etc, so maybe postgres 
could be worked in there.


--

Thanks,

Matt Riedemann

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][ceilometer][postgresql][gate][telemetry] PostgreSQL gate failure (again)

[Julien Danjou]

Hi there,

So the Ceilometer gate has been broken again by Nova today because of a
SQL bug in the placement API with PostgreSQL. This is the second time
that something like that happened: last November¹, we also reported a
bug in Nova wrt PostgreSQL.

Fortunately, Mehdi investigated and sent a fix:

  https://review.openstack.org/#/c/427667/

So the issue is going to be resolved soon. I had hope this would not
happen again, but it seems Nova does not care about PostgreSQL since
August².

I think I can understand that Nova does not want to support PostgreSQL
and therefore does not use it in the gate, however, I don't think it's a
rich idea to have other projects (e.g. Ceilometer) doing the gate check
for Nova.

My questions are simple and can be organized in a tree:

   Does Nova want to support PostgreSQL?
 /  \
   Yes  No
   /  \
   Why is there no gate? Ok, we'll have to remove the 
 PostgreSQL job in Ceilometer


Thanks!


¹  http://lists.openstack.org/pipermail/openstack-dev/2016-November/107907.html

²  http://lists.openstack.org/pipermail/openstack-dev/2016-August/101892.html

-- 
Julien Danjou
;; Free Software hacker
;; https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev