This has been discussed before apparently http://mail-archives.apache.org/mod_mbox/mesos-user/201401.mbox/%3ccaaodhhf4cvcsjfj5zusuahblw+0iie5arhpmhjvkucvjmqt...@mail.gmail.com%3E
I think that this topic will become more important now that external containerization is out. The write-outside-sandbox pattern won't work in a chroot or docker AFAIK. In addition the docker pattern for persistent data storage is to use a data-only docker image. Not sure if this is appropriate here. On Jun 26, 2014 12:42 PM, "Maxime Brugidou" <[email protected]> wrote: > There is clearly a need for persistent storage management in Mesos from > what I can observe. > > The current sandbox is what I consider ephemeral storage since it gets > lost when task exits. It can recover after a slave failure using the > recovery mechanism but for example it won't survive a slave reboot. > > Other frameworks I know of that seem to use or need persistent storage are > Cassandra and Kafka. I wonder what has been done in the framework to > survive a DC power outage for example. Is all data lost? > > As Vinod said if we want to implement persistent storage by ourselves we > need to track the resource "manually" using attributes or zk. This "trick" > will be reimplemented over and over by frameworks and will be outside > Mesos' control (I don't even know if this trick is feasible with docker > containerization). > > The proper way would be to have a persistent disk resource type (or > something else equivalent) that let you keep data on disk. The resource > will belong to a user/framework and we can have quotas. I have no idea how > to implement that since I'm not familiar with the details but it could be > using simple FS quotas and directories in the mesos directory itself (so we > mutualize ephemeral and persistent storage), it could also be on the form > of raw storage using LVM volumes to enable other sort of applications... Or > it could be both actually, mesos could have a raw volume group to use for > any sort of temporary/ephemeral and persistent volumes. > > This is probably very complex since you will need tools to report the > storage usage and do some cleanup (or have a TTL/expiry mechanism). But I > believe that every storage framework will reinvent this every time outside > Mesos. > On Jun 26, 2014 1:01 AM, "Vinod Kone" <[email protected]> wrote: > >> Thanks for listing this out Adam. >> >> Data Residency: >>> - Should we destroy the sandbox/hdfs-data when shutting down a DN? >>> - If starting DN on node that was previously running a DN, can/should we >>> try to revive the existing data? >>> >> >> I think this is one of the key challenges for a production quality HDFS >> on Mesos. Currently, since sandbox is deleted after a task exits, if all >> the data nodes that hold a block (and its replicas) get lost/killed for >> whatever reason there would be data loss. A short terms solution would be >> to write outside sandbox and use slave attributes to track where to >> re-launch data node tasks. >> >> >>
