There are two primary problems with running Ceph monitors on controllers: 1) A Ceph bug can cause the ceph-mon process to run away (overload the CPU or otherwise impact the system), causing crm to fail the whole controller node with all the other services. When ceph-mon is isolated to a dedicated host, ceph-mon failures have significantly smaller chance of disrupting other services in the environment.
2) Scaling ceph-mon cluster should follow storage requirements and shouldn't be coupled with scaling OpenStack services. -DmitryB On Mon, Feb 3, 2014 at 10:03 PM, Roman Alekseenkov <[email protected]> wrote: > Based on our experience with Ceph, I would say it should be role-based. > Meaning, option #2. > > We made a decision to always deploy Ceph monitors on controllers (that would > be an equivalent of option #1) and this decision turned out not to be the > right one. Dmitry can shed more light on this, I think. > > Roman > > > On Mon, Feb 3, 2014 at 9:44 PM, Mike Scherbakov <[email protected]> > wrote: >> >> +David, Nadya >> >> Hi Max, >> as we discussed verbally there is a major concern behind - about placement >> of MongoDB. As I understand right, it is expected that there is a huge disk >> IO consumption in case of larger deployments (let's say >50 nodes). >> If it is the case, then I would agree that we may not want to use shared >> disks for MongoDB and other OpenStack components. I see two options here: >> >> Make sure MongoDB uses dedicated disk(s) on the server where it's >> installed, and it can be part of existing controller role then >> >> Nailgun can make default allocation in a way that MongoDB has dedicated >> disk by default, if there is more than one disk on the server (which is 100% >> of real cases, I assume) >> User's experience would be simply to enable Ceilometer installation by >> clicking on checkbox. In simple mode, ceilometer + mongo will be installed >> on controller node. In HA mode, ceilometer + mongo will be installed on all >> 3 controllers under pacemaker control >> >> Make sure MongoDB is installed on a separated server >> >> In UI, user will have to: >> >> enable ceilometer checkbox ("Install Ceilometer") >> don't forget to add "ceilometer-db" (mongodb) role to one of the >> unallocated nodes >> >> UI must ensure that this role should not intersect with any other >> UI must ensure that this role is assigned to at least one node in the env, >> if ceilometer checkbox is enabled >> UI must ensure that this role is assigned to at least 3 different >> unallocated nodes in case of HA deployment mode, to ensure that we will have >> Ceilometer HA (we can skip this, but add logic that if we have more than one >> mongo - we must build cluster) >> >> In terms of simplicity and ease of use, I would vote for option #1, while >> leaving ability to place MongoDB on a separate server via Fuel CLI for >> customized deployments. #1 solves the issue with disk IO by providing >> dedicated disk(s). >> >> > Do we need HA Cluster with non-HA Mongo? >> For consistency over Fuel story, I vote for HA for all OpenStack >> components, if HA mode is chosen. So my opinion is no - we do not need such >> a case. >> >> > Puppet manifest are finished >> Great! Please pull request them asap - we will need time for reviews. I >> hope we can complete manifests for HA story by the end of the week too. >> >> Thanks, >> >> >> On Mon, Feb 3, 2014 at 5:37 PM, Max Mazur <[email protected]> wrote: >>> >>> Hi! >>> >>> >>> I'd like to add to Fuel the following options: >>> >>> 1. Simple install >>> - Ceilometer with MongoDB or Ceilometer wit If customer selected Mongo >>> it is necessary to deploy one more node with MongoDB >>> Puppet manifest are finished >>> >>> 2. HA Mode >>> - Ceilometer with MongoDB replica set. In this case we need 3 MongoDB >>> nodes to build HA replica set. >>> Puppet manifests are in progress now >>> >>> >>> Do we need HA Cluster with non-HA Mongo? >>> >>> Best Regards, >>> Max Mazur >>> >>> >>> >>> -- >>> Mailing list: https://launchpad.net/~fuel-dev >>> Post to : [email protected] >>> Unsubscribe : https://launchpad.net/~fuel-dev >>> More help : https://help.launchpad.net/ListHelp >> >> >> >> >> -- >> Mike Scherbakov >> #mihgen >> >> -- >> Mailing list: https://launchpad.net/~fuel-dev >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~fuel-dev >> More help : https://help.launchpad.net/ListHelp >> > > > -- > Mailing list: https://launchpad.net/~fuel-dev > Post to : [email protected] > Unsubscribe : https://launchpad.net/~fuel-dev > More help : https://help.launchpad.net/ListHelp > -- Dmitry Borodaenko -- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
