Re: [openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB

[Thierry Carrez]

Ronan-Alexandre Cherrueau wrote:
> Hi folks,
> 
> Recently in the Inria's Discovery initiative[1], we got in touch with
> CockroachLabs guys with an idea: make Keystone supports CockorachDB. So
> we give it a try and you can find a very first result on our GitHub[2].
> The GitHub project consists of a Vagrant file that spawns a VM with a
> CockroachDB database and then installs Keystone with Devstack using
> CockroachDB as backend.
> 
> CockroachDB claims to support the PostgreSQL protocol. It also provides
> support for SQLAlchemy that mostly inherits from the PostgreSQL one. So,
> making Keystone working with CockroachDB should be easy peasy right?
> Almost! The rest of this email describes what we have done, to make it
> works.
> [...]
Thanks Ronan-Alexandre for the update! Great to see some progress in the
experimentation there.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB

[Davanum Srinivas]

Here's a previous effort at alembic:
https://review.openstack.org/#/c/150057/

-- Dims

On Wed, Sep 6, 2017 at 11:39 AM, Michael Bayer  wrote:
> On Mon, Sep 4, 2017 at 12:06 PM, Ronan-Alexandre Cherrueau
>  wrote:
>> Hi folks,
>>
>> Recently in the Inria's Discovery initiative[1], we got in touch with
>> CockroachLabs guys with an idea: make Keystone supports CockorachDB. So
>> we give it a try and you can find a very first result on our GitHub[2].
>> The GitHub project consists of a Vagrant file that spawns a VM with a
>> CockroachDB database and then installs Keystone with Devstack using
>> CockroachDB as backend.
>>
>> CockroachDB claims to support the PostgreSQL protocol. It also provides
>> support for SQLAlchemy that mostly inherits from the PostgreSQL one. So,
>> making Keystone working with CockroachDB should be easy peasy right?
>> Almost! The rest of this email describes what we have done, to make it
>> works.
>>
>>
>> sqlalchemy-migrate doesn't work with CockroachDB
>> 
>>
>> Keystone uses a database migration tool named sqlalchemy-migrate[3].
>> This tool is called during the deployment of Keystone to migrate the
>> database from its initial versions to its actual version.
>> Unfortunately, migration failed with the following (partial)
>> stacktrace:
>>
>> ,
>> | DEBUG migrate.versioning.repository [-] Config:
>> OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'),
>> ('repository_id', 'keystone'), ('version_table', 'migrate_version'),
>> ('required_dbs', '[]'), ('use_timestamp_numbering', 'False')]))])
>> __init__ 
>> /usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83
>> | INFO migrate.versioning.api [-] 66 -> 67...
>> | CRITICAL keystone [-] KeyError: 'cockroachdb'
>> | ...
>> | TRACE keystone visitorcallable = get_engine_visitor(engine, 
>> visitor_name)
>> | TRACE keystone   File
>> "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py",
>> line 47, in get_engine_visitor
>> | TRACE keystone return get_dialect_visitor(engine.dialect, name)
>> | TRACE keystone   File
>> "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py",
>> line 62, in get_dialect_visitor
>> | TRACE keystone migrate_dialect_cls = DIALECTS[sa_dialect_name]
>> | TRACE keystone KeyError: 'cockroachdb'
>> `
>>
>> As we understand it, sqlalchemy-migrate contains dedicated SQL backend
>> codes and there is no such code for CockroachDB. As a workaround, we
>> have performed a second OS deployment with PostgreSQL as backend and
>> made a dump of the database after migrations. We then bypassed
>> migrations in our first deployment by setting up CockroachDB with the
>> dumped script. Note that we had to edit the pgdump script a little bit
>> because some features are missing in CockroachDB.
>>
>> The workaround we have to go with represents an obstacle to test
>> CockroachDB with other OpenStack core services. We would be grateful
>> if some of you would help us with adding the support of CockroachDB in
>> sqlalchemy-migrate.
>
> Please don't pursue sqlalchemy-migrate, it is an ancient and
> unmaintained project.   Keystone should be migrated to Alembic
> migrations and this would be a much more useful way to spend the
> efforts.  Alembic is fully extensible as well and I am highly
> motivated to make it work as needed.
>
>
>>
>>
>> oslo.db contains backend specific code for error handling
>> =
>>
>> The `oslo.db' library contains one file with backend-specific codes
>> (`oslo_db/sqlalchemy/exc_filters.py'[4]). This file aims at
>> abstracting database exceptions that differ but target the same class
>> of error (because each backend produces a specific exception with a
>> specific error message). Intercepted exceptions are wrapped into a
>> common OS exception to abstract backend errors in the rest of
>> `oslo.db'. CockroachDB doesn't produce same errors than PostgreSQL, so
>> we have to update this class. Note that our POC is not exhaustive
>> since we only added error messages we saw during Tempest/Rally tests.
>>
>> You can look at the differences between OpenStack/oslo.db/stable/pike
>> and our fork on GitHub[5]. We only add two lines!
>>
>>
>> CockroachDB manages isolation without lock
>> ==
>>
>> CockroachDB lets you write transactions that respect ACID properties.
>> Regarding the "I" (Isolation), CockroachDB offers snapshot and
>> serializable isolation but doesn't rely on locks to do that. So every
>> time there is concurrent editing transactions that end in a conflict,
>> then you have to retry the transactions. Fortunately, `oslo.db'
>> already offers a decorator[6] to do that automatically. But, based on
>> tests we run with Tempest/Rally, we figured out that some Keystone's
>> SQL requests needed the decorator.

Re: [openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB

[Michael Bayer]

On Mon, Sep 4, 2017 at 12:06 PM, Ronan-Alexandre Cherrueau
 wrote:
> Hi folks,
>
> Recently in the Inria's Discovery initiative[1], we got in touch with
> CockroachLabs guys with an idea: make Keystone supports CockorachDB. So
> we give it a try and you can find a very first result on our GitHub[2].
> The GitHub project consists of a Vagrant file that spawns a VM with a
> CockroachDB database and then installs Keystone with Devstack using
> CockroachDB as backend.
>
> CockroachDB claims to support the PostgreSQL protocol. It also provides
> support for SQLAlchemy that mostly inherits from the PostgreSQL one. So,
> making Keystone working with CockroachDB should be easy peasy right?
> Almost! The rest of this email describes what we have done, to make it
> works.
>
>
> sqlalchemy-migrate doesn't work with CockroachDB
> 
>
> Keystone uses a database migration tool named sqlalchemy-migrate[3].
> This tool is called during the deployment of Keystone to migrate the
> database from its initial versions to its actual version.
> Unfortunately, migration failed with the following (partial)
> stacktrace:
>
> ,
> | DEBUG migrate.versioning.repository [-] Config:
> OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'),
> ('repository_id', 'keystone'), ('version_table', 'migrate_version'),
> ('required_dbs', '[]'), ('use_timestamp_numbering', 'False')]))])
> __init__ 
> /usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83
> | INFO migrate.versioning.api [-] 66 -> 67...
> | CRITICAL keystone [-] KeyError: 'cockroachdb'
> | ...
> | TRACE keystone visitorcallable = get_engine_visitor(engine, 
> visitor_name)
> | TRACE keystone   File
> "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py",
> line 47, in get_engine_visitor
> | TRACE keystone return get_dialect_visitor(engine.dialect, name)
> | TRACE keystone   File
> "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py",
> line 62, in get_dialect_visitor
> | TRACE keystone migrate_dialect_cls = DIALECTS[sa_dialect_name]
> | TRACE keystone KeyError: 'cockroachdb'
> `
>
> As we understand it, sqlalchemy-migrate contains dedicated SQL backend
> codes and there is no such code for CockroachDB. As a workaround, we
> have performed a second OS deployment with PostgreSQL as backend and
> made a dump of the database after migrations. We then bypassed
> migrations in our first deployment by setting up CockroachDB with the
> dumped script. Note that we had to edit the pgdump script a little bit
> because some features are missing in CockroachDB.
>
> The workaround we have to go with represents an obstacle to test
> CockroachDB with other OpenStack core services. We would be grateful
> if some of you would help us with adding the support of CockroachDB in
> sqlalchemy-migrate.

Please don't pursue sqlalchemy-migrate, it is an ancient and
unmaintained project.   Keystone should be migrated to Alembic
migrations and this would be a much more useful way to spend the
efforts.  Alembic is fully extensible as well and I am highly
motivated to make it work as needed.


>
>
> oslo.db contains backend specific code for error handling
> =
>
> The `oslo.db' library contains one file with backend-specific codes
> (`oslo_db/sqlalchemy/exc_filters.py'[4]). This file aims at
> abstracting database exceptions that differ but target the same class
> of error (because each backend produces a specific exception with a
> specific error message). Intercepted exceptions are wrapped into a
> common OS exception to abstract backend errors in the rest of
> `oslo.db'. CockroachDB doesn't produce same errors than PostgreSQL, so
> we have to update this class. Note that our POC is not exhaustive
> since we only added error messages we saw during Tempest/Rally tests.
>
> You can look at the differences between OpenStack/oslo.db/stable/pike
> and our fork on GitHub[5]. We only add two lines!
>
>
> CockroachDB manages isolation without lock
> ==
>
> CockroachDB lets you write transactions that respect ACID properties.
> Regarding the "I" (Isolation), CockroachDB offers snapshot and
> serializable isolation but doesn't rely on locks to do that. So every
> time there is concurrent editing transactions that end in a conflict,
> then you have to retry the transactions. Fortunately, `oslo.db'
> already offers a decorator[6] to do that automatically. But, based on
> tests we run with Tempest/Rally, we figured out that some Keystone's
> SQL requests needed the decorator.
>
> You can look at the differences between OpenStack/keystone/stable/pike
> and our fork on GitHub[7].
>
>
> What's next?
> 
>
> You can drop yourself on the VM as a stack user and run Rally tests
> (see README). Tempest is disabled because we have an 

Re: [openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB

[Lance Bragstad]

On 09/04/2017 11:06 AM, Ronan-Alexandre Cherrueau wrote:
> Hi folks,
>
> Recently in the Inria's Discovery initiative[1], we got in touch with
> CockroachLabs guys with an idea: make Keystone supports CockorachDB. So
> we give it a try and you can find a very first result on our GitHub[2].
> The GitHub project consists of a Vagrant file that spawns a VM with a
> CockroachDB database and then installs Keystone with Devstack using
> CockroachDB as backend.
>
> CockroachDB claims to support the PostgreSQL protocol. It also provides
> support for SQLAlchemy that mostly inherits from the PostgreSQL one. So,
> making Keystone working with CockroachDB should be easy peasy right?
> Almost! The rest of this email describes what we have done, to make it
> works.
>
>
> sqlalchemy-migrate doesn't work with CockroachDB
> 
>
> Keystone uses a database migration tool named sqlalchemy-migrate[3].
> This tool is called during the deployment of Keystone to migrate the
> database from its initial versions to its actual version.
> Unfortunately, migration failed with the following (partial)
> stacktrace:
>
> ,
> | DEBUG migrate.versioning.repository [-] Config:
> OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'),
> ('repository_id', 'keystone'), ('version_table', 'migrate_version'),
> ('required_dbs', '[]'), ('use_timestamp_numbering', 'False')]))])
> __init__ 
> /usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83
> | INFO migrate.versioning.api [-] 66 -> 67...
> | CRITICAL keystone [-] KeyError: 'cockroachdb'
> | ...
> | TRACE keystone visitorcallable = get_engine_visitor(engine, 
> visitor_name)
> | TRACE keystone   File
> "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py",
> line 47, in get_engine_visitor
> | TRACE keystone return get_dialect_visitor(engine.dialect, name)
> | TRACE keystone   File
> "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py",
> line 62, in get_dialect_visitor
> | TRACE keystone migrate_dialect_cls = DIALECTS[sa_dialect_name]
> | TRACE keystone KeyError: 'cockroachdb'
> `
>
> As we understand it, sqlalchemy-migrate contains dedicated SQL backend
> codes and there is no such code for CockroachDB. As a workaround, we
> have performed a second OS deployment with PostgreSQL as backend and
> made a dump of the database after migrations. We then bypassed
> migrations in our first deployment by setting up CockroachDB with the
> dumped script. Note that we had to edit the pgdump script a little bit
> because some features are missing in CockroachDB.
>
> The workaround we have to go with represents an obstacle to test
> CockroachDB with other OpenStack core services. We would be grateful
> if some of you would help us with adding the support of CockroachDB in
> sqlalchemy-migrate.
>
>
> oslo.db contains backend specific code for error handling
> =
>
> The `oslo.db' library contains one file with backend-specific codes
> (`oslo_db/sqlalchemy/exc_filters.py'[4]). This file aims at
> abstracting database exceptions that differ but target the same class
> of error (because each backend produces a specific exception with a
> specific error message). Intercepted exceptions are wrapped into a
> common OS exception to abstract backend errors in the rest of
> `oslo.db'. CockroachDB doesn't produce same errors than PostgreSQL, so
> we have to update this class. Note that our POC is not exhaustive
> since we only added error messages we saw during Tempest/Rally tests.
>
> You can look at the differences between OpenStack/oslo.db/stable/pike
> and our fork on GitHub[5]. We only add two lines!
>
>
> CockroachDB manages isolation without lock
> ==
>
> CockroachDB lets you write transactions that respect ACID properties.
> Regarding the "I" (Isolation), CockroachDB offers snapshot and
> serializable isolation but doesn't rely on locks to do that. So every
> time there is concurrent editing transactions that end in a conflict,
> then you have to retry the transactions. Fortunately, `oslo.db'
> already offers a decorator[6] to do that automatically. But, based on
> tests we run with Tempest/Rally, we figured out that some Keystone's
> SQL requests needed the decorator.
>
> You can look at the differences between OpenStack/keystone/stable/pike
> and our fork on GitHub[7].

Correct me if I'm wrong, but wouldn't this be useful in keystone
regardless of using CockroachDB? It looks like we use the same decorator
in a couple other places in keystone already [0]. Would you mind
proposing this through gerrit directly? It seems like it would be useful
to iterate on it in review. Unless there is a bug exposed in
stable/pike, I don't think we'll be able to back port it. Getting it
into master seems like a good thing, though.

[0] https://review.openstack.org/#/c/416872/

>

[openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB

[Ronan-Alexandre Cherrueau]

Hi folks,

Recently in the Inria's Discovery initiative[1], we got in touch with
CockroachLabs guys with an idea: make Keystone supports CockorachDB. So
we give it a try and you can find a very first result on our GitHub[2].
The GitHub project consists of a Vagrant file that spawns a VM with a
CockroachDB database and then installs Keystone with Devstack using
CockroachDB as backend.

CockroachDB claims to support the PostgreSQL protocol. It also provides
support for SQLAlchemy that mostly inherits from the PostgreSQL one. So,
making Keystone working with CockroachDB should be easy peasy right?
Almost! The rest of this email describes what we have done, to make it
works.


sqlalchemy-migrate doesn't work with CockroachDB


Keystone uses a database migration tool named sqlalchemy-migrate[3].
This tool is called during the deployment of Keystone to migrate the
database from its initial versions to its actual version.
Unfortunately, migration failed with the following (partial)
stacktrace:

,
| DEBUG migrate.versioning.repository [-] Config:
OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'),
('repository_id', 'keystone'), ('version_table', 'migrate_version'),
('required_dbs', '[]'), ('use_timestamp_numbering', 'False')]))])
__init__ 
/usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83
| INFO migrate.versioning.api [-] 66 -> 67...
| CRITICAL keystone [-] KeyError: 'cockroachdb'
| ...
| TRACE keystone visitorcallable = get_engine_visitor(engine, visitor_name)
| TRACE keystone   File
"/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py",
line 47, in get_engine_visitor
| TRACE keystone return get_dialect_visitor(engine.dialect, name)
| TRACE keystone   File
"/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py",
line 62, in get_dialect_visitor
| TRACE keystone migrate_dialect_cls = DIALECTS[sa_dialect_name]
| TRACE keystone KeyError: 'cockroachdb'
`

As we understand it, sqlalchemy-migrate contains dedicated SQL backend
codes and there is no such code for CockroachDB. As a workaround, we
have performed a second OS deployment with PostgreSQL as backend and
made a dump of the database after migrations. We then bypassed
migrations in our first deployment by setting up CockroachDB with the
dumped script. Note that we had to edit the pgdump script a little bit
because some features are missing in CockroachDB.

The workaround we have to go with represents an obstacle to test
CockroachDB with other OpenStack core services. We would be grateful
if some of you would help us with adding the support of CockroachDB in
sqlalchemy-migrate.


oslo.db contains backend specific code for error handling
=

The `oslo.db' library contains one file with backend-specific codes
(`oslo_db/sqlalchemy/exc_filters.py'[4]). This file aims at
abstracting database exceptions that differ but target the same class
of error (because each backend produces a specific exception with a
specific error message). Intercepted exceptions are wrapped into a
common OS exception to abstract backend errors in the rest of
`oslo.db'. CockroachDB doesn't produce same errors than PostgreSQL, so
we have to update this class. Note that our POC is not exhaustive
since we only added error messages we saw during Tempest/Rally tests.

You can look at the differences between OpenStack/oslo.db/stable/pike
and our fork on GitHub[5]. We only add two lines!


CockroachDB manages isolation without lock
==

CockroachDB lets you write transactions that respect ACID properties.
Regarding the "I" (Isolation), CockroachDB offers snapshot and
serializable isolation but doesn't rely on locks to do that. So every
time there is concurrent editing transactions that end in a conflict,
then you have to retry the transactions. Fortunately, `oslo.db'
already offers a decorator[6] to do that automatically. But, based on
tests we run with Tempest/Rally, we figured out that some Keystone's
SQL requests needed the decorator.

You can look at the differences between OpenStack/keystone/stable/pike
and our fork on GitHub[7].


What's next?


You can drop yourself on the VM as a stack user and run Rally tests
(see README). Tempest is disabled because we have an issue with its
installation[8]. Note that modifications we made to make it works are
minimal. This is promising for the adoption of CockroachDB by other
core services. Nonetheless:

- Fixing the problem with sqlalchemy-migrate will be helpful to ease
  the deployment process. If you are willing to help, we will be
  really grateful.

- We have to look through the performance prism: Keystone over
  CockroachDB vs. Keystone over Galera. This part may involve more
  modifications of `oslo.db'. One thing we have in mind is the
  management of retrying transactions since