Re: [openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB
Ronan-Alexandre Cherrueau wrote: > Hi folks, > > Recently in the Inria's Discovery initiative[1], we got in touch with > CockroachLabs guys with an idea: make Keystone supports CockorachDB. So > we give it a try and you can find a very first result on our GitHub[2]. > The GitHub project consists of a Vagrant file that spawns a VM with a > CockroachDB database and then installs Keystone with Devstack using > CockroachDB as backend. > > CockroachDB claims to support the PostgreSQL protocol. It also provides > support for SQLAlchemy that mostly inherits from the PostgreSQL one. So, > making Keystone working with CockroachDB should be easy peasy right? > Almost! The rest of this email describes what we have done, to make it > works. > [...] Thanks Ronan-Alexandre for the update! Great to see some progress in the experimentation there. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB
Here's a previous effort at alembic: https://review.openstack.org/#/c/150057/ -- Dims On Wed, Sep 6, 2017 at 11:39 AM, Michael Bayer wrote: > On Mon, Sep 4, 2017 at 12:06 PM, Ronan-Alexandre Cherrueau > wrote: >> Hi folks, >> >> Recently in the Inria's Discovery initiative[1], we got in touch with >> CockroachLabs guys with an idea: make Keystone supports CockorachDB. So >> we give it a try and you can find a very first result on our GitHub[2]. >> The GitHub project consists of a Vagrant file that spawns a VM with a >> CockroachDB database and then installs Keystone with Devstack using >> CockroachDB as backend. >> >> CockroachDB claims to support the PostgreSQL protocol. It also provides >> support for SQLAlchemy that mostly inherits from the PostgreSQL one. So, >> making Keystone working with CockroachDB should be easy peasy right? >> Almost! The rest of this email describes what we have done, to make it >> works. >> >> >> sqlalchemy-migrate doesn't work with CockroachDB >> >> >> Keystone uses a database migration tool named sqlalchemy-migrate[3]. >> This tool is called during the deployment of Keystone to migrate the >> database from its initial versions to its actual version. >> Unfortunately, migration failed with the following (partial) >> stacktrace: >> >> , >> | DEBUG migrate.versioning.repository [-] Config: >> OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), >> ('repository_id', 'keystone'), ('version_table', 'migrate_version'), >> ('required_dbs', '[]'), ('use_timestamp_numbering', 'False')]))]) >> __init__ >> /usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83 >> | INFO migrate.versioning.api [-] 66 -> 67... >> | CRITICAL keystone [-] KeyError: 'cockroachdb' >> | ... >> | TRACE keystone visitorcallable = get_engine_visitor(engine, >> visitor_name) >> | TRACE keystone File >> "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", >> line 47, in get_engine_visitor >> | TRACE keystone return get_dialect_visitor(engine.dialect, name) >> | TRACE keystone File >> "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", >> line 62, in get_dialect_visitor >> | TRACE keystone migrate_dialect_cls = DIALECTS[sa_dialect_name] >> | TRACE keystone KeyError: 'cockroachdb' >> ` >> >> As we understand it, sqlalchemy-migrate contains dedicated SQL backend >> codes and there is no such code for CockroachDB. As a workaround, we >> have performed a second OS deployment with PostgreSQL as backend and >> made a dump of the database after migrations. We then bypassed >> migrations in our first deployment by setting up CockroachDB with the >> dumped script. Note that we had to edit the pgdump script a little bit >> because some features are missing in CockroachDB. >> >> The workaround we have to go with represents an obstacle to test >> CockroachDB with other OpenStack core services. We would be grateful >> if some of you would help us with adding the support of CockroachDB in >> sqlalchemy-migrate. > > Please don't pursue sqlalchemy-migrate, it is an ancient and > unmaintained project. Keystone should be migrated to Alembic > migrations and this would be a much more useful way to spend the > efforts. Alembic is fully extensible as well and I am highly > motivated to make it work as needed. > > >> >> >> oslo.db contains backend specific code for error handling >> = >> >> The `oslo.db' library contains one file with backend-specific codes >> (`oslo_db/sqlalchemy/exc_filters.py'[4]). This file aims at >> abstracting database exceptions that differ but target the same class >> of error (because each backend produces a specific exception with a >> specific error message). Intercepted exceptions are wrapped into a >> common OS exception to abstract backend errors in the rest of >> `oslo.db'. CockroachDB doesn't produce same errors than PostgreSQL, so >> we have to update this class. Note that our POC is not exhaustive >> since we only added error messages we saw during Tempest/Rally tests. >> >> You can look at the differences between OpenStack/oslo.db/stable/pike >> and our fork on GitHub[5]. We only add two lines! >> >> >> CockroachDB manages isolation without lock >> == >> >> CockroachDB lets you write transactions that respect ACID properties. >> Regarding the "I" (Isolation), CockroachDB offers snapshot and >> serializable isolation but doesn't rely on locks to do that. So every >> time there is concurrent editing transactions that end in a conflict, >> then you have to retry the transactions. Fortunately, `oslo.db' >> already offers a decorator[6] to do that automatically. But, based on >> tests we run with Tempest/Rally, we figured out that some Keystone's >> SQL requests needed the decorator. >> >> You can look at the differences between OpenStack/
Re: [openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB
On Mon, Sep 4, 2017 at 12:06 PM, Ronan-Alexandre Cherrueau wrote: > Hi folks, > > Recently in the Inria's Discovery initiative[1], we got in touch with > CockroachLabs guys with an idea: make Keystone supports CockorachDB. So > we give it a try and you can find a very first result on our GitHub[2]. > The GitHub project consists of a Vagrant file that spawns a VM with a > CockroachDB database and then installs Keystone with Devstack using > CockroachDB as backend. > > CockroachDB claims to support the PostgreSQL protocol. It also provides > support for SQLAlchemy that mostly inherits from the PostgreSQL one. So, > making Keystone working with CockroachDB should be easy peasy right? > Almost! The rest of this email describes what we have done, to make it > works. > > > sqlalchemy-migrate doesn't work with CockroachDB > > > Keystone uses a database migration tool named sqlalchemy-migrate[3]. > This tool is called during the deployment of Keystone to migrate the > database from its initial versions to its actual version. > Unfortunately, migration failed with the following (partial) > stacktrace: > > , > | DEBUG migrate.versioning.repository [-] Config: > OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), > ('repository_id', 'keystone'), ('version_table', 'migrate_version'), > ('required_dbs', '[]'), ('use_timestamp_numbering', 'False')]))]) > __init__ > /usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83 > | INFO migrate.versioning.api [-] 66 -> 67... > | CRITICAL keystone [-] KeyError: 'cockroachdb' > | ... > | TRACE keystone visitorcallable = get_engine_visitor(engine, > visitor_name) > | TRACE keystone File > "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", > line 47, in get_engine_visitor > | TRACE keystone return get_dialect_visitor(engine.dialect, name) > | TRACE keystone File > "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", > line 62, in get_dialect_visitor > | TRACE keystone migrate_dialect_cls = DIALECTS[sa_dialect_name] > | TRACE keystone KeyError: 'cockroachdb' > ` > > As we understand it, sqlalchemy-migrate contains dedicated SQL backend > codes and there is no such code for CockroachDB. As a workaround, we > have performed a second OS deployment with PostgreSQL as backend and > made a dump of the database after migrations. We then bypassed > migrations in our first deployment by setting up CockroachDB with the > dumped script. Note that we had to edit the pgdump script a little bit > because some features are missing in CockroachDB. > > The workaround we have to go with represents an obstacle to test > CockroachDB with other OpenStack core services. We would be grateful > if some of you would help us with adding the support of CockroachDB in > sqlalchemy-migrate. Please don't pursue sqlalchemy-migrate, it is an ancient and unmaintained project. Keystone should be migrated to Alembic migrations and this would be a much more useful way to spend the efforts. Alembic is fully extensible as well and I am highly motivated to make it work as needed. > > > oslo.db contains backend specific code for error handling > = > > The `oslo.db' library contains one file with backend-specific codes > (`oslo_db/sqlalchemy/exc_filters.py'[4]). This file aims at > abstracting database exceptions that differ but target the same class > of error (because each backend produces a specific exception with a > specific error message). Intercepted exceptions are wrapped into a > common OS exception to abstract backend errors in the rest of > `oslo.db'. CockroachDB doesn't produce same errors than PostgreSQL, so > we have to update this class. Note that our POC is not exhaustive > since we only added error messages we saw during Tempest/Rally tests. > > You can look at the differences between OpenStack/oslo.db/stable/pike > and our fork on GitHub[5]. We only add two lines! > > > CockroachDB manages isolation without lock > == > > CockroachDB lets you write transactions that respect ACID properties. > Regarding the "I" (Isolation), CockroachDB offers snapshot and > serializable isolation but doesn't rely on locks to do that. So every > time there is concurrent editing transactions that end in a conflict, > then you have to retry the transactions. Fortunately, `oslo.db' > already offers a decorator[6] to do that automatically. But, based on > tests we run with Tempest/Rally, we figured out that some Keystone's > SQL requests needed the decorator. > > You can look at the differences between OpenStack/keystone/stable/pike > and our fork on GitHub[7]. > > > What's next? > > > You can drop yourself on the VM as a stack user and run Rally tests > (see README). Tempest is disabled because we have an issue with its > installation[8]. Note th
Re: [openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB
On 09/04/2017 11:06 AM, Ronan-Alexandre Cherrueau wrote: > Hi folks, > > Recently in the Inria's Discovery initiative[1], we got in touch with > CockroachLabs guys with an idea: make Keystone supports CockorachDB. So > we give it a try and you can find a very first result on our GitHub[2]. > The GitHub project consists of a Vagrant file that spawns a VM with a > CockroachDB database and then installs Keystone with Devstack using > CockroachDB as backend. > > CockroachDB claims to support the PostgreSQL protocol. It also provides > support for SQLAlchemy that mostly inherits from the PostgreSQL one. So, > making Keystone working with CockroachDB should be easy peasy right? > Almost! The rest of this email describes what we have done, to make it > works. > > > sqlalchemy-migrate doesn't work with CockroachDB > > > Keystone uses a database migration tool named sqlalchemy-migrate[3]. > This tool is called during the deployment of Keystone to migrate the > database from its initial versions to its actual version. > Unfortunately, migration failed with the following (partial) > stacktrace: > > , > | DEBUG migrate.versioning.repository [-] Config: > OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), > ('repository_id', 'keystone'), ('version_table', 'migrate_version'), > ('required_dbs', '[]'), ('use_timestamp_numbering', 'False')]))]) > __init__ > /usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83 > | INFO migrate.versioning.api [-] 66 -> 67... > | CRITICAL keystone [-] KeyError: 'cockroachdb' > | ... > | TRACE keystone visitorcallable = get_engine_visitor(engine, > visitor_name) > | TRACE keystone File > "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", > line 47, in get_engine_visitor > | TRACE keystone return get_dialect_visitor(engine.dialect, name) > | TRACE keystone File > "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", > line 62, in get_dialect_visitor > | TRACE keystone migrate_dialect_cls = DIALECTS[sa_dialect_name] > | TRACE keystone KeyError: 'cockroachdb' > ` > > As we understand it, sqlalchemy-migrate contains dedicated SQL backend > codes and there is no such code for CockroachDB. As a workaround, we > have performed a second OS deployment with PostgreSQL as backend and > made a dump of the database after migrations. We then bypassed > migrations in our first deployment by setting up CockroachDB with the > dumped script. Note that we had to edit the pgdump script a little bit > because some features are missing in CockroachDB. > > The workaround we have to go with represents an obstacle to test > CockroachDB with other OpenStack core services. We would be grateful > if some of you would help us with adding the support of CockroachDB in > sqlalchemy-migrate. > > > oslo.db contains backend specific code for error handling > = > > The `oslo.db' library contains one file with backend-specific codes > (`oslo_db/sqlalchemy/exc_filters.py'[4]). This file aims at > abstracting database exceptions that differ but target the same class > of error (because each backend produces a specific exception with a > specific error message). Intercepted exceptions are wrapped into a > common OS exception to abstract backend errors in the rest of > `oslo.db'. CockroachDB doesn't produce same errors than PostgreSQL, so > we have to update this class. Note that our POC is not exhaustive > since we only added error messages we saw during Tempest/Rally tests. > > You can look at the differences between OpenStack/oslo.db/stable/pike > and our fork on GitHub[5]. We only add two lines! > > > CockroachDB manages isolation without lock > == > > CockroachDB lets you write transactions that respect ACID properties. > Regarding the "I" (Isolation), CockroachDB offers snapshot and > serializable isolation but doesn't rely on locks to do that. So every > time there is concurrent editing transactions that end in a conflict, > then you have to retry the transactions. Fortunately, `oslo.db' > already offers a decorator[6] to do that automatically. But, based on > tests we run with Tempest/Rally, we figured out that some Keystone's > SQL requests needed the decorator. > > You can look at the differences between OpenStack/keystone/stable/pike > and our fork on GitHub[7]. Correct me if I'm wrong, but wouldn't this be useful in keystone regardless of using CockroachDB? It looks like we use the same decorator in a couple other places in keystone already [0]. Would you mind proposing this through gerrit directly? It seems like it would be useful to iterate on it in review. Unless there is a bug exposed in stable/pike, I don't think we'll be able to back port it. Getting it into master seems like a good thing, though. [0] https://review.openstack.org/#/c/416872/ >
[openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB
Hi folks, Recently in the Inria's Discovery initiative[1], we got in touch with CockroachLabs guys with an idea: make Keystone supports CockorachDB. So we give it a try and you can find a very first result on our GitHub[2]. The GitHub project consists of a Vagrant file that spawns a VM with a CockroachDB database and then installs Keystone with Devstack using CockroachDB as backend. CockroachDB claims to support the PostgreSQL protocol. It also provides support for SQLAlchemy that mostly inherits from the PostgreSQL one. So, making Keystone working with CockroachDB should be easy peasy right? Almost! The rest of this email describes what we have done, to make it works. sqlalchemy-migrate doesn't work with CockroachDB Keystone uses a database migration tool named sqlalchemy-migrate[3]. This tool is called during the deployment of Keystone to migrate the database from its initial versions to its actual version. Unfortunately, migration failed with the following (partial) stacktrace: , | DEBUG migrate.versioning.repository [-] Config: OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), ('repository_id', 'keystone'), ('version_table', 'migrate_version'), ('required_dbs', '[]'), ('use_timestamp_numbering', 'False')]))]) __init__ /usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83 | INFO migrate.versioning.api [-] 66 -> 67... | CRITICAL keystone [-] KeyError: 'cockroachdb' | ... | TRACE keystone visitorcallable = get_engine_visitor(engine, visitor_name) | TRACE keystone File "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", line 47, in get_engine_visitor | TRACE keystone return get_dialect_visitor(engine.dialect, name) | TRACE keystone File "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", line 62, in get_dialect_visitor | TRACE keystone migrate_dialect_cls = DIALECTS[sa_dialect_name] | TRACE keystone KeyError: 'cockroachdb' ` As we understand it, sqlalchemy-migrate contains dedicated SQL backend codes and there is no such code for CockroachDB. As a workaround, we have performed a second OS deployment with PostgreSQL as backend and made a dump of the database after migrations. We then bypassed migrations in our first deployment by setting up CockroachDB with the dumped script. Note that we had to edit the pgdump script a little bit because some features are missing in CockroachDB. The workaround we have to go with represents an obstacle to test CockroachDB with other OpenStack core services. We would be grateful if some of you would help us with adding the support of CockroachDB in sqlalchemy-migrate. oslo.db contains backend specific code for error handling = The `oslo.db' library contains one file with backend-specific codes (`oslo_db/sqlalchemy/exc_filters.py'[4]). This file aims at abstracting database exceptions that differ but target the same class of error (because each backend produces a specific exception with a specific error message). Intercepted exceptions are wrapped into a common OS exception to abstract backend errors in the rest of `oslo.db'. CockroachDB doesn't produce same errors than PostgreSQL, so we have to update this class. Note that our POC is not exhaustive since we only added error messages we saw during Tempest/Rally tests. You can look at the differences between OpenStack/oslo.db/stable/pike and our fork on GitHub[5]. We only add two lines! CockroachDB manages isolation without lock == CockroachDB lets you write transactions that respect ACID properties. Regarding the "I" (Isolation), CockroachDB offers snapshot and serializable isolation but doesn't rely on locks to do that. So every time there is concurrent editing transactions that end in a conflict, then you have to retry the transactions. Fortunately, `oslo.db' already offers a decorator[6] to do that automatically. But, based on tests we run with Tempest/Rally, we figured out that some Keystone's SQL requests needed the decorator. You can look at the differences between OpenStack/keystone/stable/pike and our fork on GitHub[7]. What's next? You can drop yourself on the VM as a stack user and run Rally tests (see README). Tempest is disabled because we have an issue with its installation[8]. Note that modifications we made to make it works are minimal. This is promising for the adoption of CockroachDB by other core services. Nonetheless: - Fixing the problem with sqlalchemy-migrate will be helpful to ease the deployment process. If you are willing to help, we will be really grateful. - We have to look through the performance prism: Keystone over CockroachDB vs. Keystone over Galera. This part may involve more modifications of `oslo.db'. One thing we have in mind is the management of retrying transactions since Cockroac