Sorry for the spam, silence for a week then there's no shutting me up :) Apart from registering the osd bootstrap key, I now have a running 3 node cluster
2015-03-03 07:01:40.484418 mon.0 [INF] pgmap v20: 64 pgs: 64 active+clean; 0 bytes data, 101 MB used, 1498 GB / 1498 GB avail On Mon, Mar 2, 2015 at 10:57 PM, Stuart Fox <[email protected]> wrote: > Update 2 > > Running > > * ceph auth del client.bootstrap-osd* > > Then > > *ceph auth add client.bootstrap-osd -i > /var/lib/ceph/bootstrap-osd/ceph.keyring* > > Allows me to run puppet and get an osd working > > On Mon, Mar 2, 2015 at 10:37 PM, Stuart Fox <[email protected]> wrote: > >> Sorry again ..... its been a trying week at work and I've only just had >> time to pick this up >> >> So, progress. I can now get the mons up and running. Took some fiddling >> with, the $cluster param breaks just about everything so had to stick with >> the default cluster name. I can live with that! I havent had to patch >> anything this time round. >> >> Next problem seems to be registering the osd_bootstrap key, the error is >> >> Notice: >> /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns: >> + ceph --name mon. --keyring /var/lib/ceph/mon/ceph-ceph-001/keyring auth >> add client.bootstrap-osd --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring >> >> Notice: >> /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns: >> libust[28614/28614]: Warning: HOME environment variable not set. Disabling >> LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305) >> >> Notice: >> /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns: >> Error EINVAL: entity client.bootstrap-osd exists but key does not match >> >> Error: /bin/true # comment to satisfy puppet syntax requirements >> >> set -ex >> >> ceph --name 'mon.' --keyring >> '/var/lib/ceph/mon/ceph-ceph-001/keyring' auth add client.bootstrap-osd >> --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring returned 22 instead of >> one of [0] >> >> Error: >> /Stage[main]/Openfish::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns: >> change from notrun to 0 failed: /bin/true # comment to satisfy puppet >> syntax requirements >> >> set -ex >> >> ceph --name 'mon.' --keyring >> '/var/lib/ceph/mon/ceph-ceph-001/keyring' auth add client.bootstrap-osd >> --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring returned 22 instead of >> one of [0] >> >> >> I havent gotten as far as the osd's or gateways yet >> >> Help greatly appreciated! >> >> >> >> On Tue, Feb 24, 2015 at 10:31 AM, David Gurtner <[email protected]> >> wrote: >> >>> Hi again >>> >>> Just to test if the modules generally work on Trusty I set up a three >>> node cluster with shared mon/osd using the configuration below and >>> that worked. I was using the master branch of the module and had >>> firewall disabled. >>> >>> Cheers, >>> David >>> >>> >>> $admin_key = 'AQCTg71RsNIHORAAW+O6FCMZWBjmVfMIPk3MhQ==' >>> $mon_key = 'AQDesGZSsC7KJBAAw+W/Z4eGSQGAIbxWjxjvfw==' >>> $bootstrap_osd_key = 'AQABsWZSgEDmJhAAkAGSOOAJwrMHrM5Pz5On1A==' >>> $fsid = '066F558C-6789-4A93-AAF1-5AF1BA01A3AD' >>> >>> node /first|second|third/ { >>> class { 'ceph::repo': } >>> class { 'ceph': >>> fsid => $fsid, >>> mon_initial_members => 'first,second,third', >>> mon_host => '10.11.12.2,10.11.12.3,10.11.12.4', >>> } >>> ceph::mon { $::hostname: >>> key => $mon_key, >>> } >>> Ceph::Key { >>> inject => true, >>> inject_as_id => 'mon.', >>> inject_keyring => "/var/lib/ceph/mon/ceph-${::hostname}/keyring", >>> } >>> ceph::key { 'client.admin': >>> secret => $admin_key, >>> cap_mon => 'allow *', >>> cap_osd => 'allow *', >>> cap_mds => 'allow', >>> } >>> ceph::key { 'client.bootstrap-osd': >>> secret => $bootstrap_osd_key, >>> keyring_path => '/var/lib/ceph/bootstrap-osd/ceph.keyring', >>> cap_mon => 'allow profile bootstrap-osd', >>> } >>> ceph::osd { '/dev/sdb': } >>> } >>> >>> >>> On Mon, Feb 23, 2015 at 11:39 PM, David Gurtner <[email protected]> >>> wrote: >>> > Hi Stuart >>> > >>> > osds can only come up once at least the initial mon services are >>> > running and formed a quorum, so lets concentrate at getting the mons >>> > up first. >>> > If using cephx authentication all mon services need to be started at >>> > the same time (a similar time), translated to puppet this means all >>> > the puppet runs on the mons need to be started within a certain time >>> > frame (before the timeout of the mons trying to connect each other). >>> > The reason being that the mons first need to form a quorum before most >>> > of the further actions can happen. >>> > >>> > Can you provide some additional information about your setup? >>> Specifically: >>> > - what's your exact puppet modules configuration >>> > - what version are you running? >>> > - please be aware that running on Ubuntu 14.04 is currently not >>> > getting integration tested (currently only Ubuntu 12.04 and CentOS 6 >>> > are tested, support for CentOS 7 is in the works). >>> > - how many mons are you running >>> > - are the osds running on the same hosts as the mons >>> > - do you use a single network, or do you use separate cluster/public >>> networks >>> > - do you have a firewall running, if yes, can you share your firewall >>> > configuration (iptables -vnL) >>> > - what exactly did you need to patch? >>> > >>> > Cheers, >>> > David >>> > >>> > On Mon, Feb 23, 2015 at 8:16 PM, Stuart Fox <[email protected]> >>> wrote: >>> >> Sorry to ask a question and run, I had a busy weekend. I'll be back >>> on the >>> >> ceph POC later today. >>> >> >>> >> I'll paste some outputs from the puppet runs shortly but essentially, >>> no mon >>> >> or osd comes up. >>> >> I had to patch several manifests to explicitly reference >>> /etc/ceph/ceph.conf >>> >> and given that the puppet module calls a reasonable amount of shell >>> code, >>> >> getting debug info is not easy. The puppet runs exit correctly but no >>> >> service comes up. >>> >> I'm using an almost cut'n'paste from USECASES.md, even down to >>> keeping the >>> >> keys the same. >>> >> >>> >> BR, >>> >> Stuart >>> >> >>> >> On Sun, Feb 22, 2015 at 6:57 AM, David Moreau Simard < >>> [email protected]> >>> >> wrote: >>> >>> >>> >>> Hey Stuart, >>> >>> >>> >>> You might want to look at (and use, if you want) the built-in roles >>> and >>> >>> profiles layer [1] which gives you a good idea of how the module is >>> used. >>> >>> >>> >>> It leverages ceph::profile::params to pass values [2] to other >>> classes >>> >>> such as ceph::profile::osd [3]. >>> >>> >>> >>> Another good place to look at how to use the module would be the >>> >>> integration tests [4]. >>> >>> Since the module is integration tested, this means we actually use >>> the >>> >>> module to deploy a virtual Ceph cluster and we test that the cluster >>> works - >>> >>> see for example the tests for ceph::profile::osd [4]. >>> >>> >>> >>> I'm dmsimard on #puppet-openstack and #openstack-operators if you >>> need a >>> >>> hand to get this to work. >>> >>> This is still a relatively new module in comparison to the likes of >>> >>> puppet-nova and such. Feedback and contributions are appreciated! >>> >>> >>> >>> [1] >>> >>> >>> https://github.com/stackforge/puppet-ceph/tree/master/manifests/profile >>> >>> [2] >>> >>> >>> https://github.com/stackforge/puppet-ceph/blob/master/manifests/profile/params.pp >>> >>> [3] >>> >>> >>> https://github.com/stackforge/puppet-ceph/blob/master/manifests/profile/osd.pp >>> >>> [4] >>> >>> >>> https://github.com/stackforge/puppet-ceph/blob/master/spec/system/ceph_profile_osd_spec.rb >>> >>> -- >>> >>> David Moreau Simard >>> >>> >>> >>> From: Andrew Woodward <[email protected]> >>> >>> Date: Saturday, February 21, 2015 at 4:39 PM >>> >>> To: Stuart Fox <[email protected]> >>> >>> Cc: "[email protected]" >>> >>> <[email protected]>, " >>> [email protected]" >>> >>> <[email protected]> >>> >>> Subject: Re: [Openstack-operators] Ceph puppet module >>> >>> >>> >>> [+puppet-openstack ml] >>> >>> >>> >>> Stuart, >>> >>> >>> >>> Please review the USECASES.md such as [1]. You should find usable >>> examples >>> >>> there. If you are still having problems please reach out with a more >>> detail >>> >>> about what you configuration you are attempting to deploy. The module >>> >>> authors can be found in #puppet-openstack and are usually lurking on >>> the >>> >>> puppet-openstack mailing list (CC'd). >>> >>> >>> >>> >>> >>> [1] >>> https://github.com/stackforge/puppet-ceph/blob/master/USECASES.md#i-want-to-operate-a-production-cluster >>> >>> >>> >>> >>> >>> >>> >>> On Sat, Feb 21, 2015 at 12:04 AM, Stuart Fox <[email protected]> >>> wrote: >>> >>>> >>> >>>> Hey all >>> >>>> >>> >>>> Im having a complete nightmare trying to get ceph deployed using the >>> >>>> https://github.com/stackforge/puppet-ceph module. Scant >>> documentation isn't >>> >>>> helping my cause! >>> >>>> >>> >>>> Is anybody else using this module? Im attempting to deploy ceph as a >>> >>>> cinder backend in Juno on Ubuntu 14.04 although I haven't gotten as >>> far as >>> >>>> integration yet. >>> >>>> >>> >>>> Does anyone have working example's I could look at? >>> >>>> >>> >>>> -- >>> >>>> BR, >>> >>>> Stuart >>> >>>> >>> >>>> _______________________________________________ >>> >>>> OpenStack-operators mailing list >>> >>>> [email protected] >>> >>>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >>> >>>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Andrew >>> >>> Mirantis >>> >>> Fuel community ambassador >>> >>> Ceph community >>> >> >>> >> >>> >> >>> >> >>> >> -- >>> >> BR, >>> >> Stuart >>> >> >>> >> _______________________________________________ >>> >> OpenStack-operators mailing list >>> >> [email protected] >>> >> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >>> >> >>> >> >> >> >> -- >> BR, >> Stuart >> > > > > -- > BR, > Stuart > -- BR, Stuart
_______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
