Hi Loic, Thanks for the reply! I started implementing these first using ceph-deploy, but ran into lots of issues because there is no upstart daemon running within the containers. So while ceph-deploy would successfully log into the containers remotely (I had to run sshd in the containers), it would fail when it tried to start the daemons. I spent a few days hitting a wall with this before I found Seán McCord's repository at https://github.com/ulexus/docker-ceph which configures/starts everything without ceph-deploy, and went that route. So our containers are based on his work, but we added a lot of service coordination/discovery within CoreOS.
I'm still planning on writing a blog post about our use of Ceph in containers, but haven't found the time. I'm hoping to get to it next week. In the meantime, let me know if you have any questions about the implementation! Chris *Chris Armstrong*Head of Services & Community OpDemand / Deis.io GitHub: https://github.com/deis/deis -- Docs: http://docs.deis.io/ On Thu, Oct 30, 2014 at 5:36 AM, Hunter Nield <[email protected]> wrote: > Great idea Loic! > > I'd forgotten about the storage-driver side but is a great fit with Ceph > > On Thu, Oct 30, 2014 at 6:50 PM, Loic Dachary <[email protected]> wrote: > >> Hi, >> >> It would also be great to have a Ceph docker storage driver. >> >> https://github.com/docker/docker/issues/8854 >> >> Cheers >> >> On 30/10/2014 11:06, Hunter Nield wrote: >> > Great to see this discussion starting. There is work being done in this >> repo for Ceph in Docker - https://github.com/Ulexus/docker-ceph >> > >> > Independently of this, we're using RBD as backing for the Docker >> containers but still installing Ceph as part of the system and mounting >> outside of the container before launch. Ceph in Docker is in the backlog >> for us but becoming important. >> > >> > There is a proposal for volume drivers in Docker which would also be a >> great fit for RBD (via fuse?). Hopefully it gets moving soon - >> https://github.com/docker/docker/issues/7249 >> > >> > >> > >> > On Thu, Oct 30, 2014 at 5:18 PM, Loic Dachary <[email protected] >> <mailto:[email protected]>> wrote: >> > >> > Hi Christopher, >> > >> > Very interesting setup :-) Last week-end I discussed this in theory >> with Johan Euphrosine and did not know you had something already. Deploying >> a mon in a container is fairly straightforward and I wonder if the boot >> script >> > >> > https://github.com/deis/deis/blob/master/store/monitor/bin/boot >> > >> > could somehow be coupled to ceph-deploy to benefit from its >> flexibility. Deploying an OSD with >> > >> > https://github.com/deis/deis/blob/master/store/daemon/bin/boot >> > >> > is more complicated and I don't quite understand how it uses the >> udev rules and their relationship with the upstart / systemd scripts that >> play an important role in the OSD lifecycle >> > >> > https://github.com/ceph/ceph/tree/giant/udev >> > https://github.com/ceph/ceph/tree/giant/src/upstart >> > https://github.com/ceph/ceph/blob/giant/src/upstart/ceph-osd.conf >> > >> > Cheers >> > >> > On 30/10/2014 06:16, Christopher Armstrong wrote: >> > > Hey Patrick, >> > > >> > > We recently added a new component to Deis which is based entirely >> on running Ceph in containers. We're running mons, OSDs, and MDSes in >> containers, and consuming from containers with radosgw as well as CephFS. >> See the source here: https://github.com/deis/deis/tree/master/store >> > > >> > > I'm pretty proud of the work, and would be more than happy to >> write a blog post about it if you'd like. >> > > >> > > Chris >> > > >> > > *Chris Armstrong >> > > *Head of Services >> > > OpDemand / Deis.io >> > > >> > > GitHub: https://github.com/deis/deis -- Docs: >> http://docs.deis.io/ >> > > >> > > >> > > On Wed, Oct 29, 2014 at 8:26 PM, Patrick McGarry < >> [email protected] <mailto:[email protected]> <mailto: >> [email protected] <mailto:[email protected]>>> wrote: >> > > >> > > Hey cephers, >> > > >> > > Given some of the recent interest in utilizing Docker with >> Ceph I'm >> > > taking another survey of the landscape. I know that Loic >> recently got >> > > Teuthology running with Docker (http://dachary.org/?p=3330) >> but I'd >> > > like to look at running a containerized Ceph setup as well. >> > > >> > > So far I see that Sebastien did an experiment back in 2013: >> > > >> > > >> http://www.sebastien-han.fr/blog/2013/09/19/how-I-barely-got-my-first-ceph-mon-running-in-docker/ >> > > >> > > and Lorieri had a CoreOS experiment: >> > > >> > > >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-August/042063.html >> > > >> > > but there have been a few people that mentioned their >> experiments to >> > > me in passing at cons and other places. I'd love to gather >> any >> > > experience that people have gleaned in this area by >> aggregating blog >> > > entries and other notes. So, if you have a successful >> Ceph+Docker >> > > setup and would be willing to write a short doc via >> email/blog/wiki I >> > > would greatly appreciate it. Thanks! >> > > >> > > >> > > Best Regards, >> > > >> > > Patrick McGarry >> > > Director Ceph Community || Red Hat >> > > http://ceph.com || http://community.redhat.com >> > > @scuttlemonkey || @ceph >> > > _______________________________________________ >> > > ceph-users mailing list >> > > [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > >> > > >> > > >> > > >> > > _______________________________________________ >> > > ceph-users mailing list >> > > [email protected] <mailto:[email protected]> >> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > >> > >> > -- >> > Loïc Dachary, Artisan Logiciel Libre >> > >> > >> > _______________________________________________ >> > ceph-users mailing list >> > [email protected] <mailto:[email protected]> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > >> > >> >> -- >> Loïc Dachary, Artisan Logiciel Libre >> >> > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
