Public bug reported: This is similar to older bug https://bugs.launchpad.net/bugs/1543937 but in 'nova-manage db purge'. Purging a large dataset causes this failure in Galera cluster. Mariadb log has following error when this happens:
2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648 2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2 This happens because the transaction is too large for mariadb to handle. 'nova-manage db archive_deleted_rows' works around this by limiting the number of max_rows using --max_rows argument and checking it against db.MAX_INT variable, we might have to do similar thing for purge Traceback: nova-manage --config-file /etc/nova/nova.conf db purge --verbose --all-cells --all An error has occurred: Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 2793, in main ret = fn(*fn_args, **fn_kwargs) File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 454, in purge status_fn=status) File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 4426, in purge_shadow_tables deleted = conn.execute(delete) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute return meth(self, multiparams, params) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement distilled_params, File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context e, statement, parameters, cursor, context File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception util.raise_(newraise, with_traceback=exc_info[2], from_=e) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1301, in _execute_context self._root._commit_impl(autocommit=True) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 773, in _commit_impl self._handle_dbapi_exception(e, None, None, None, None) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception util.raise_(newraise, with_traceback=exc_info[2], from_=e) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 771, in _commit_impl self.engine.dialect.do_commit(self.connection) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 2463, in do_commit dbapi_connection.commit() File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 422, in commit self._read_ok_packet() File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 396, in _read_ok_packet pkt = self._read_packet() File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet packet.raise_for_error() File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error err.raise_mysql_exception(self._data) File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception raise errorclass(errno, errval) sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1180, 'Got error 90 "Message too long" during COMMIT') (Background on this error at: http://sqlalche.me/e/13/e3q8) MariaDB log has following error: 2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648 2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2 More information on how to replicate this and workaround this using mariadb is available here https://www.percona.com/blog/2015/10/26/how-big-can-your-galera-transactions-be/ ** Affects: nova Importance: Undecided Status: New ** Tags: nova-manage -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1983188 Title: nova-manage db purge fails on large datasets Status in OpenStack Compute (nova): New Bug description: This is similar to older bug https://bugs.launchpad.net/bugs/1543937 but in 'nova-manage db purge'. Purging a large dataset causes this failure in Galera cluster. Mariadb log has following error when this happens: 2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648 2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2 This happens because the transaction is too large for mariadb to handle. 'nova-manage db archive_deleted_rows' works around this by limiting the number of max_rows using --max_rows argument and checking it against db.MAX_INT variable, we might have to do similar thing for purge Traceback: nova-manage --config-file /etc/nova/nova.conf db purge --verbose --all-cells --all An error has occurred: Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 2793, in main ret = fn(*fn_args, **fn_kwargs) File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/cmd/manage.py", line 454, in purge status_fn=status) File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 4426, in purge_shadow_tables deleted = conn.execute(delete) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute return meth(self, multiparams, params) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement distilled_params, File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context e, statement, parameters, cursor, context File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception util.raise_(newraise, with_traceback=exc_info[2], from_=e) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1301, in _execute_context self._root._commit_impl(autocommit=True) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 773, in _commit_impl self._handle_dbapi_exception(e, None, None, None, None) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1508, in _handle_dbapi_exception util.raise_(newraise, with_traceback=exc_info[2], from_=e) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 771, in _commit_impl self.engine.dialect.do_commit(self.connection) File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 2463, in do_commit dbapi_connection.commit() File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 422, in commit self._read_ok_packet() File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 396, in _read_ok_packet pkt = self._read_packet() File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 676, in _read_packet packet.raise_for_error() File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/protocol.py", line 223, in raise_for_error err.raise_mysql_exception(self._data) File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/err.py", line 107, in raise_mysql_exception raise errorclass(errno, errval) sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1180, 'Got error 90 "Message too long" during COMMIT') (Background on this error at: http://sqlalche.me/e/13/e3q8) MariaDB log has following error: 2022-07-30 20:22:10 1567 [Warning] WSREP: transaction size limit (2147483647) exceeded: 2147483648 2022-07-30 20:22:10 1567 [ERROR] WSREP: rbr write fail, data_len: 0, 2 More information on how to replicate this and workaround this using mariadb is available here https://www.percona.com/blog/2015/10/26/how-big-can-your-galera-transactions-be/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1983188/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp