There is clearly a need for persistent storage management in Mesos from what I can observe.
The current sandbox is what I consider ephemeral storage since it gets lost when task exits. It can recover after a slave failure using the recovery mechanism but for example it won't survive a slave reboot. Other frameworks I know of that seem to use or need persistent storage are Cassandra and Kafka. I wonder what has been done in the framework to survive a DC power outage for example. Is all data lost? As Vinod said if we want to implement persistent storage by ourselves we need to track the resource "manually" using attributes or zk. This "trick" will be reimplemented over and over by frameworks and will be outside Mesos' control (I don't even know if this trick is feasible with docker containerization). The proper way would be to have a persistent disk resource type (or something else equivalent) that let you keep data on disk. The resource will belong to a user/framework and we can have quotas. I have no idea how to implement that since I'm not familiar with the details but it could be using simple FS quotas and directories in the mesos directory itself (so we mutualize ephemeral and persistent storage), it could also be on the form of raw storage using LVM volumes to enable other sort of applications... Or it could be both actually, mesos could have a raw volume group to use for any sort of temporary/ephemeral and persistent volumes. This is probably very complex since you will need tools to report the storage usage and do some cleanup (or have a TTL/expiry mechanism). But I believe that every storage framework will reinvent this every time outside Mesos. On Jun 26, 2014 1:01 AM, "Vinod Kone" <[email protected]> wrote: > Thanks for listing this out Adam. > > Data Residency: >> - Should we destroy the sandbox/hdfs-data when shutting down a DN? >> - If starting DN on node that was previously running a DN, can/should we >> try to revive the existing data? >> > > I think this is one of the key challenges for a production quality HDFS on > Mesos. Currently, since sandbox is deleted after a task exits, if all the > data nodes that hold a block (and its replicas) get lost/killed for > whatever reason there would be data loss. A short terms solution would be > to write outside sandbox and use slave attributes to track where to > re-launch data node tasks. > > >
