Will do. We haven't really had any stability issues once the containers are running - they are the same daemons that would run on your host OS, they just happen to be in a container. We have seen a few cases during startup where there are placement groups stuck inactive/peering, which prevents the metadata servers from coming up. Restarting the daemon containers resolves this, although I haven't figured out why it's occurring.
*Chris Armstrong*Head of Services & Community OpDemand / Deis.io GitHub: https://github.com/deis/deis -- Docs: http://docs.deis.io/ On Fri, Nov 14, 2014 at 3:31 PM, Robert LeBlanc <[email protected]> wrote: > Ceph in Docker is very intriguing to me, but I understood that there were > still a number of stability and implementation issues. What is your > experience? Please post a link to your blog when you are done, I'd be > interested in reading it. > > On Fri, Nov 14, 2014 at 4:15 PM, Christopher Armstrong <[email protected] > > wrote: > >> Hi Loic, >> >> Thanks for the reply! I started implementing these first using >> ceph-deploy, but ran into lots of issues because there is no upstart daemon >> running within the containers. So while ceph-deploy would successfully log >> into the containers remotely (I had to run sshd in the containers), it >> would fail when it tried to start the daemons. I spent a few days hitting a >> wall with this before I found Seán McCord's repository at >> https://github.com/ulexus/docker-ceph which configures/starts everything >> without ceph-deploy, and went that route. So our containers are based on >> his work, but we added a lot of service coordination/discovery within >> CoreOS. >> >> I'm still planning on writing a blog post about our use of Ceph in >> containers, but haven't found the time. I'm hoping to get to it next week. >> In the meantime, let me know if you have any questions about the >> implementation! >> >> Chris >> >> >> *Chris Armstrong*Head of Services & Community >> OpDemand / Deis.io >> >> GitHub: https://github.com/deis/deis -- Docs: http://docs.deis.io/ >> >> >> On Thu, Oct 30, 2014 at 5:36 AM, Hunter Nield <[email protected]> wrote: >> >>> Great idea Loic! >>> >>> I'd forgotten about the storage-driver side but is a great fit with Ceph >>> >>> On Thu, Oct 30, 2014 at 6:50 PM, Loic Dachary <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> It would also be great to have a Ceph docker storage driver. >>>> >>>> https://github.com/docker/docker/issues/8854 >>>> >>>> Cheers >>>> >>>> On 30/10/2014 11:06, Hunter Nield wrote: >>>> > Great to see this discussion starting. There is work being done in >>>> this repo for Ceph in Docker - https://github.com/Ulexus/docker-ceph >>>> > >>>> > Independently of this, we're using RBD as backing for the Docker >>>> containers but still installing Ceph as part of the system and mounting >>>> outside of the container before launch. Ceph in Docker is in the backlog >>>> for us but becoming important. >>>> > >>>> > There is a proposal for volume drivers in Docker which would also be >>>> a great fit for RBD (via fuse?). Hopefully it gets moving soon - >>>> https://github.com/docker/docker/issues/7249 >>>> > >>>> > >>>> > >>>> > On Thu, Oct 30, 2014 at 5:18 PM, Loic Dachary <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> > >>>> > Hi Christopher, >>>> > >>>> > Very interesting setup :-) Last week-end I discussed this in >>>> theory with Johan Euphrosine and did not know you had something already. >>>> Deploying a mon in a container is fairly straightforward and I wonder if >>>> the boot script >>>> > >>>> > https://github.com/deis/deis/blob/master/store/monitor/bin/boot >>>> > >>>> > could somehow be coupled to ceph-deploy to benefit from its >>>> flexibility. Deploying an OSD with >>>> > >>>> > https://github.com/deis/deis/blob/master/store/daemon/bin/boot >>>> > >>>> > is more complicated and I don't quite understand how it uses the >>>> udev rules and their relationship with the upstart / systemd scripts that >>>> play an important role in the OSD lifecycle >>>> > >>>> > https://github.com/ceph/ceph/tree/giant/udev >>>> > https://github.com/ceph/ceph/tree/giant/src/upstart >>>> > https://github.com/ceph/ceph/blob/giant/src/upstart/ceph-osd.conf >>>> > >>>> > Cheers >>>> > >>>> > On 30/10/2014 06:16, Christopher Armstrong wrote: >>>> > > Hey Patrick, >>>> > > >>>> > > We recently added a new component to Deis which is based >>>> entirely on running Ceph in containers. We're running mons, OSDs, and MDSes >>>> in containers, and consuming from containers with radosgw as well as >>>> CephFS. See the source here: >>>> https://github.com/deis/deis/tree/master/store >>>> > > >>>> > > I'm pretty proud of the work, and would be more than happy to >>>> write a blog post about it if you'd like. >>>> > > >>>> > > Chris >>>> > > >>>> > > *Chris Armstrong >>>> > > *Head of Services >>>> > > OpDemand / Deis.io >>>> > > >>>> > > GitHub: https://github.com/deis/deis -- Docs: >>>> http://docs.deis.io/ >>>> > > >>>> > > >>>> > > On Wed, Oct 29, 2014 at 8:26 PM, Patrick McGarry < >>>> [email protected] <mailto:[email protected]> <mailto: >>>> [email protected] <mailto:[email protected]>>> wrote: >>>> > > >>>> > > Hey cephers, >>>> > > >>>> > > Given some of the recent interest in utilizing Docker with >>>> Ceph I'm >>>> > > taking another survey of the landscape. I know that Loic >>>> recently got >>>> > > Teuthology running with Docker (http://dachary.org/?p=3330) >>>> but I'd >>>> > > like to look at running a containerized Ceph setup as well. >>>> > > >>>> > > So far I see that Sebastien did an experiment back in 2013: >>>> > > >>>> > > >>>> http://www.sebastien-han.fr/blog/2013/09/19/how-I-barely-got-my-first-ceph-mon-running-in-docker/ >>>> > > >>>> > > and Lorieri had a CoreOS experiment: >>>> > > >>>> > > >>>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-August/042063.html >>>> > > >>>> > > but there have been a few people that mentioned their >>>> experiments to >>>> > > me in passing at cons and other places. I'd love to gather >>>> any >>>> > > experience that people have gleaned in this area by >>>> aggregating blog >>>> > > entries and other notes. So, if you have a successful >>>> Ceph+Docker >>>> > > setup and would be willing to write a short doc via >>>> email/blog/wiki I >>>> > > would greatly appreciate it. Thanks! >>>> > > >>>> > > >>>> > > Best Regards, >>>> > > >>>> > > Patrick McGarry >>>> > > Director Ceph Community || Red Hat >>>> > > http://ceph.com || http://community.redhat.com >>>> > > @scuttlemonkey || @ceph >>>> > > _______________________________________________ >>>> > > ceph-users mailing list >>>> > > [email protected] <mailto:[email protected]> >>>> <mailto:[email protected] <mailto:[email protected]>> >>>> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > _______________________________________________ >>>> > > ceph-users mailing list >>>> > > [email protected] <mailto:[email protected]> >>>> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> > > >>>> > >>>> > -- >>>> > Loïc Dachary, Artisan Logiciel Libre >>>> > >>>> > >>>> > _______________________________________________ >>>> > ceph-users mailing list >>>> > [email protected] <mailto:[email protected]> >>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> > >>>> > >>>> >>>> -- >>>> Loïc Dachary, Artisan Logiciel Libre >>>> >>>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> [email protected] >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
