Re: [openstack-dev] Sahara Job Binaries Storage
Hi Trevor, Just revisiting this, has there been any progress to deprecate sahara jobs -> internal db mechanism? and/or the config option to disable internal db storage? Regards, Jerico > On 18 Mar 2016, at 12:55 AM, Trevor McKay <tmc...@redhat.com> wrote: > > Hi Jerico, > > Internal db storage for job binaries was added at > the start of EDP as an alternative for sites that do > not have swift running. Since then, we've also added > integration with manila so that job binaries can be > stored in manila shares. > > You are correct, storing lots of binaries in the > sahara db could make the database grow very large. > Swift or manila should be used for production, internal > storage is a good option for development/test. > > There is currently no way to disable internal storage. > We can took a look at adding such an option -- in fact > we have talked informally about the possibility of > deprecating internal db storage since swift and manila > are both mature at this point. We should discuss that > at the upcoming summit. > > Best, > > Trevor > > On Thu, 2016-03-17 at 10:27 +1100, Jerico Revote wrote: >> Hello, >> >> >> When deploying Sahara, Sahara docos suggests to >> increase max_allowed_packet to 256MB, >> for internal database storing of job binaries. >> There could be hundreds of job binaries to be uploaded/created into >> Sahara, >> which would then cause the database to grow as well. >> Does anyone using Sahara encountered database sizing issues using >> internal db storage? >> >> >> It looks like swift is the more logical place for storing job >> binaries >> (in our case we have a global swift cluster), and this is also >> available to the user. >> Is there a way to only enable the swift way for storing job binaries? >> >> Thanks, >> >> >> Jerico >> >> >> >> >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Deleting a cluster in Sahara SQL/PyMYSQL Error
Yes, first, when creating a new cluster, it get stuck on "validating", then tried deleting it but get stuck again on "deleting", and seeing that SQL error. Will submit a bug @ Launchpad. On 25 March 2016 at 01:56, Mike Bayer <mba...@redhat.com> wrote: > Id recommend filing a bug in Launchpad against Sahara for that. Can you > reproduce it ? > > > > > On 03/23/2016 07:10 PM, Jerico Revote wrote: > >> Hello, >> >> When trying to delete a cluster in sahara, >> I'm getting the following error: >> >> code 500 and message 'Internal Server Error' >>> 2016-03-23 17:25:21.651 18827 ERROR sahara.utils.api >>> [req-d797bbc8-7932-4187-a428-565f9d834f8b ] Traceback (most recent >>> call last): >>> OperationalError: (pymysql.err.OperationalError) (2014, 'Command Out >>> of Sync') >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> [req-377ef364-f2c7-4343-b32c-3741bfc0a05b ] DB exception wrapped. >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> Traceback (most recent call last): >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", >>> line 1139, in _execute_context >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> context) >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", >>> line 450, in do_execute >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> cursor.execute(statement, parameters) >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 132, >>> in execute >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> result = self._query(query) >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 271, >>> in _query >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> conn.query(q) >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line >>> 726, in query >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> self._affected_rows = self._read_query_result(unbuffered=unbuffered) >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line >>> 861, in _read_query_result >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> result.read() >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line >>> 1064, in read >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> first_packet = self.connection._read_packet() >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line >>> 825, in _read_packet >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> packet = packet_type(self) >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line >>> 242, in __init__ >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> self._recv_packet(connection) >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line >>> 248, in _recv_packet >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> packet_header = connection._read_bytes(4) >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line >>> 839, in _read_bytes >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> if len(data) < num_bytes: >>> 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters >>> TypeError: object of type 'NoneType' has no len() >>> 201
[openstack-dev] Deleting a cluster in Sahara SQL/PyMYSQL Error
Hello, When trying to delete a cluster in sahara, I'm getting the following error: > code 500 and message 'Internal Server Error' > 2016-03-23 17:25:21.651 18827 ERROR sahara.utils.api > [req-d797bbc8-7932-4187-a428-565f9d834f8b ] Traceback (most recent call last): > OperationalError: (pymysql.err.OperationalError) (2014, 'Command Out of Sync') > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters > [req-377ef364-f2c7-4343-b32c-3741bfc0a05b ] DB exception wrapped. > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters Traceback > (most recent call last): > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters File > "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in > _execute_context > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters > context) > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters File > "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in > do_execute > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters > cursor.execute(statement, parameters) > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters File > "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 132, in execute > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters result > = self._query(query) > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters File > "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 271, in _query > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters > conn.query(q) > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters File > "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 726, in query > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters > self._affected_rows = self._read_query_result(unbuffered=unbuffered) > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters File > "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 861, in > _read_query_result > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters > result.read() > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters File > "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1064, in read > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters > first_packet = self.connection._read_packet() > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters File > "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 825, in > _read_packet > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters packet > = packet_type(self) > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters File > "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 242, in > __init__ > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters > self._recv_packet(connection) > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters File > "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 248, in > _recv_packet > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters > packet_header = connection._read_bytes(4) > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters File > "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 839, in > _read_bytes > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters if > len(data) < num_bytes: > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters TypeError: > object of type 'NoneType' has no len() > 2016-03-23 17:25:35.803 18823 ERROR oslo_db.sqlalchemy.exc_filters > 2016-03-23 17:25:35.808 18823 ERROR sahara.utils.api > [req-377ef364-f2c7-4343-b32c-3741bfc0a05b ] Request aborted with status code > 500 and message 'Internal Server Error' > 2016-03-23 17:25:35.809 18823 ERROR sahara.utils.api > [req-377ef364-f2c7-4343-b32c-3741bfc0a05b ] Traceback (most recent call last): > OperationalError: (pymysql.err.OperationalError) (2014, 'Command Out of Sync') Any idea what could this mean? Thanks As a result, sahara clusters are stuck in "Deleting" state. > pkg -l | grep -i sahara > ii python-sahara1:3.0.0-0ubuntu1~cloud0 all > OpenStack data processing cluster as a service - library > ii sahara-api 1:3.0.0-0ubuntu1~cloud0 all > OpenStack data processing cluster as a service - API > ii sahara-common1:3.0.0-0ubuntu1~cloud0 all > OpenStack data processing cluster as a service - common files Regards, Jerico __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
[openstack-dev] Sahara Job Binaries Storage
Hello, When deploying Sahara, Sahara docos suggests to increase max_allowed_packet to 256MB, for internal database storing of job binaries. There could be hundreds of job binaries to be uploaded/created into Sahara, which would then cause the database to grow as well. Does anyone using Sahara encountered database sizing issues using internal db storage? It looks like swift is the more logical place for storing job binaries (in our case we have a global swift cluster), and this is also available to the user. Is there a way to only enable the swift way for storing job binaries? Thanks, Jerico __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Sahara Job Binaries Storage
Hi Trevor, Thanks for the explanation, looking forward to that on/off option and/or internal db deprecation. Regards, Jerico > On 18 Mar 2016, at 12:55 AM, Trevor McKay <tmc...@redhat.com> wrote: > > Hi Jerico, > > Internal db storage for job binaries was added at > the start of EDP as an alternative for sites that do > not have swift running. Since then, we've also added > integration with manila so that job binaries can be > stored in manila shares. > > You are correct, storing lots of binaries in the > sahara db could make the database grow very large. > Swift or manila should be used for production, internal > storage is a good option for development/test. > > There is currently no way to disable internal storage. > We can took a look at adding such an option -- in fact > we have talked informally about the possibility of > deprecating internal db storage since swift and manila > are both mature at this point. We should discuss that > at the upcoming summit. > > Best, > > Trevor > > On Thu, 2016-03-17 at 10:27 +1100, Jerico Revote wrote: >> Hello, >> >> >> When deploying Sahara, Sahara docos suggests to >> increase max_allowed_packet to 256MB, >> for internal database storing of job binaries. >> There could be hundreds of job binaries to be uploaded/created into >> Sahara, >> which would then cause the database to grow as well. >> Does anyone using Sahara encountered database sizing issues using >> internal db storage? >> >> >> It looks like swift is the more logical place for storing job >> binaries >> (in our case we have a global swift cluster), and this is also >> available to the user. >> Is there a way to only enable the swift way for storing job binaries? >> >> Thanks, >> >> >> Jerico >> >> >> >> >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
