Re: [openstack-dev] Sahara Job Binaries Storage
Hi Jerico, we talked about it at Summit in one of the design sessions, but afaik there is no blueprint or spec yet. I don't see why it can't happen in Newton, however. Best, Trevor On Thu, 2016-05-26 at 16:14 +1000, Jerico Revote wrote: > Hi Trevor, > > Just revisiting this, > has there been any progress to deprecate sahara jobs -> internal db mechanism? > and/or the config option to disable internal db storage? > > Regards, > > Jerico > > > > > On 18 Mar 2016, at 12:55 AM, Trevor McKay wrote: > > > > Hi Jerico, > > > > Internal db storage for job binaries was added at > > the start of EDP as an alternative for sites that do > > not have swift running. Since then, we've also added > > integration with manila so that job binaries can be > > stored in manila shares. > > > > You are correct, storing lots of binaries in the > > sahara db could make the database grow very large. > > Swift or manila should be used for production, internal > > storage is a good option for development/test. > > > > There is currently no way to disable internal storage. > > We can took a look at adding such an option -- in fact > > we have talked informally about the possibility of > > deprecating internal db storage since swift and manila > > are both mature at this point. We should discuss that > > at the upcoming summit. > > > > Best, > > > > Trevor > > > > On Thu, 2016-03-17 at 10:27 +1100, Jerico Revote wrote: > >> Hello, > >> > >> > >> When deploying Sahara, Sahara docos suggests to > >> increase max_allowed_packet to 256MB, > >> for internal database storing of job binaries. > >> There could be hundreds of job binaries to be uploaded/created into > >> Sahara, > >> which would then cause the database to grow as well. > >> Does anyone using Sahara encountered database sizing issues using > >> internal db storage? > >> > >> > >> It looks like swift is the more logical place for storing job > >> binaries > >> (in our case we have a global swift cluster), and this is also > >> available to the user. > >> Is there a way to only enable the swift way for storing job binaries? > >> > >> Thanks, > >> > >> > >> Jerico > >> > >> > >> > >> > >> > >> > >> __ > >> OpenStack Development Mailing List (not for usage questions) > >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Sahara Job Binaries Storage
Hi Trevor, Just revisiting this, has there been any progress to deprecate sahara jobs -> internal db mechanism? and/or the config option to disable internal db storage? Regards, Jerico > On 18 Mar 2016, at 12:55 AM, Trevor McKay wrote: > > Hi Jerico, > > Internal db storage for job binaries was added at > the start of EDP as an alternative for sites that do > not have swift running. Since then, we've also added > integration with manila so that job binaries can be > stored in manila shares. > > You are correct, storing lots of binaries in the > sahara db could make the database grow very large. > Swift or manila should be used for production, internal > storage is a good option for development/test. > > There is currently no way to disable internal storage. > We can took a look at adding such an option -- in fact > we have talked informally about the possibility of > deprecating internal db storage since swift and manila > are both mature at this point. We should discuss that > at the upcoming summit. > > Best, > > Trevor > > On Thu, 2016-03-17 at 10:27 +1100, Jerico Revote wrote: >> Hello, >> >> >> When deploying Sahara, Sahara docos suggests to >> increase max_allowed_packet to 256MB, >> for internal database storing of job binaries. >> There could be hundreds of job binaries to be uploaded/created into >> Sahara, >> which would then cause the database to grow as well. >> Does anyone using Sahara encountered database sizing issues using >> internal db storage? >> >> >> It looks like swift is the more logical place for storing job >> binaries >> (in our case we have a global swift cluster), and this is also >> available to the user. >> Is there a way to only enable the swift way for storing job binaries? >> >> Thanks, >> >> >> Jerico >> >> >> >> >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Sahara Job Binaries Storage
Hello, When deploying Sahara, Sahara docos suggests to increase max_allowed_packet to 256MB, for internal database storing of job binaries. There could be hundreds of job binaries to be uploaded/created into Sahara, which would then cause the database to grow as well. Does anyone using Sahara encountered database sizing issues using internal db storage? It looks like swift is the more logical place for storing job binaries (in our case we have a global swift cluster), and this is also available to the user. Is there a way to only enable the swift way for storing job binaries? Thanks, Jerico __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Sahara Job Binaries Storage
Hi Jerico, Internal db storage for job binaries was added at the start of EDP as an alternative for sites that do not have swift running. Since then, we've also added integration with manila so that job binaries can be stored in manila shares. You are correct, storing lots of binaries in the sahara db could make the database grow very large. Swift or manila should be used for production, internal storage is a good option for development/test. There is currently no way to disable internal storage. We can took a look at adding such an option -- in fact we have talked informally about the possibility of deprecating internal db storage since swift and manila are both mature at this point. We should discuss that at the upcoming summit. Best, Trevor On Thu, 2016-03-17 at 10:27 +1100, Jerico Revote wrote: > Hello, > > > When deploying Sahara, Sahara docos suggests to > increase max_allowed_packet to 256MB, > for internal database storing of job binaries. > There could be hundreds of job binaries to be uploaded/created into > Sahara, > which would then cause the database to grow as well. > Does anyone using Sahara encountered database sizing issues using > internal db storage? > > > It looks like swift is the more logical place for storing job > binaries > (in our case we have a global swift cluster), and this is also > available to the user. > Is there a way to only enable the swift way for storing job binaries? > > Thanks, > > > Jerico > > > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Sahara Job Binaries Storage
Hi Jerico, Internal db storage for job binaries was added at the start of EDP as an alternative for sites that do not have swift running. Since then, we've also added integration with manila so that job binaries can be stored in manila shares. You are correct, storing lots of binaries in the sahara db could make the database grow very large. Swift or manila should be used for production, internal storage is a good option for development/test. There is currently no way to disable internal storage. We can took a look at adding such an option -- in fact we have talked informally about the possibility of deprecating internal db storage since swift and manila are both mature at this point. We should discuss that at the upcoming summit. Best, Trevor On Thu, 2016-03-17 at 10:27 +1100, Jerico Revote wrote: > Hello, > > > When deploying Sahara, Sahara docos suggests to > increase max_allowed_packet to 256MB, > for internal database storing of job binaries. > There could be hundreds of job binaries to be uploaded/created into > Sahara, > which would then cause the database to grow as well. > Does anyone using Sahara encountered database sizing issues using > internal db storage? > > > It looks like swift is the more logical place for storing job > binaries > (in our case we have a global swift cluster), and this is also > available to the user. > Is there a way to only enable the swift way for storing job binaries? > > Thanks, > > > Jerico > > > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Sahara Job Binaries Storage
Hi Trevor, Thanks for the explanation, looking forward to that on/off option and/or internal db deprecation. Regards, Jerico > On 18 Mar 2016, at 12:55 AM, Trevor McKay wrote: > > Hi Jerico, > > Internal db storage for job binaries was added at > the start of EDP as an alternative for sites that do > not have swift running. Since then, we've also added > integration with manila so that job binaries can be > stored in manila shares. > > You are correct, storing lots of binaries in the > sahara db could make the database grow very large. > Swift or manila should be used for production, internal > storage is a good option for development/test. > > There is currently no way to disable internal storage. > We can took a look at adding such an option -- in fact > we have talked informally about the possibility of > deprecating internal db storage since swift and manila > are both mature at this point. We should discuss that > at the upcoming summit. > > Best, > > Trevor > > On Thu, 2016-03-17 at 10:27 +1100, Jerico Revote wrote: >> Hello, >> >> >> When deploying Sahara, Sahara docos suggests to >> increase max_allowed_packet to 256MB, >> for internal database storing of job binaries. >> There could be hundreds of job binaries to be uploaded/created into >> Sahara, >> which would then cause the database to grow as well. >> Does anyone using Sahara encountered database sizing issues using >> internal db storage? >> >> >> It looks like swift is the more logical place for storing job >> binaries >> (in our case we have a global swift cluster), and this is also >> available to the user. >> Is there a way to only enable the swift way for storing job binaries? >> >> Thanks, >> >> >> Jerico >> >> >> >> >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
