Data locality in Mesos

[Tobias Pfeiffer]

Hi,

one of the major selling points of HDFS is (was?) that it is possible to
schedule a Hadoop job close to where the data that it operates on is.  I am
not using HDFS, but I was wondering if/how Mesos supports an approach to
schedule a job to a machine that has a certain file/dataset already locally
as opposed to scheduling it to a machine that would have to access it via
the network or download to the local disk first.

I was wondering if Mesos attributes could be used:  I could have an
attribute `datasets` of type `set` and then node A could have {dataset1,
dataset17, dataset3} and node B could have {dataset17, dataset5} and during
scheduling I could decide based on this attribute where to run a task.
However, I was wondering if there are dynamic changes of such attributes
possible.  Imagine that node A deletes dataset17 from the local cache and
downloads dataset5 instead, then I would like to update the `datasets`
attribute dynamically, but without affecting the jobs that are running on
node A.  Is such a thing possible?

Is there an approach other than attributes to describe the data that
resides on a node in order to achieve data locality?

Thanks
Tobias

Re: Agent Working Directory Best Practices

[Steven Schlansker]

> On Jun 26, 2017, at 5:30 PM, James Peach  wrote:
> 
> 
>> On Jun 26, 2017, at 4:05 PM, Steven Schlansker  
>> wrote:
>> 
>> 
>>> On Jun 25, 2017, at 11:24 PM, Benjamin Mahler  wrote:
>>> 
>>> As a data point, as far as I'm aware, most users are using a local work 
>>> directory, not an NFS mounted one. Would love to hear from anyone on the 
>>> list if they are doing this, and if there are any subtleties that should be 
>>> documented.
>> 
>> We don't run NFS in particular but we did originally use a SAN -- two 
>> observations:
>> 
>> NFS (historically, maybe it's better now, but doubtful...) has really bad 
>> failure modes.
>> Network failures can cause serious hangs both in user-space and 
>> kernel-space.  Such
>> hangs can be impossible to clear without rebooting the machine, and in some 
>> edge cases
>> can even make it difficult or impossible to reboot the machine via normal 
>> means.
> 
> You need to make sure to mount with the "intr" option.
> 
> https://speakerdeck.com/gnb/130-lca2008-nfs-tuning-secrets-d7

That's not without some caveats.  nfs(5):

The intr / nointr mount option is deprecated after kernel 2.6.25. Only SIGKILL 
can interrupt a pending NFS operation on these kernels, and if specified, this 
mount option is ignored to provide backwards compatibility with older kernels.
Using the intr option is preferred to using the soft option because it is 
significantly less likely to result in data corruption.

...

NB: A so-called "soft" timeout can cause silent data corruption in certain 
cases. As such, use the soft option only when client responsiveness is more 
important than data integrity. Using NFS over TCP or increasing the value of 
the retrans option may mitigate some of the risks of using the soft option.


So, 'intr' is deprecated / removed on any reasonable kernel, and 'soft' has 
silent data corruption issues.
Typical Linux, having a broken implementation that then points you to instead 
use a deprecated / removed implementation :)

I'm sure there's a way to get NFS working great.  Just pointing out that you'll 
need an expert to take ownership of it!

> 
>> 
>> Network attached drives (our SAN) are less reliable, slower, and more complex
>> (read: more failure modes) than local disk.  It's also a really big single 
>> point
>> of failure.  So far our only true cluster outages have been due to failure of
>> the SAN, since it took down all nodes at once -- once we removed the SAN, 
>> future
>> failures had islands of availability and any properly written application
>> could continue running (obviously without network resources) through the 
>> incident.
>> 
>> Maybe this isn't a huge deal for your use case, which might differ from ours.
>> For us, it was enough of a problem that we now purchase local SSD scratch 
>> space
>> for every node just so that we have some storage we can depend on a bit more
>> than network attached storage.
>> 
>>> 
>>> On Thu, Jun 22, 2017 at 11:13 PM,  wrote:
>>> Hi,
>>> 
>>> We have a couple of server nodes mainly used for computational tasks in
>>> our mesos cluster. These servers have beefy cpus, gpus etc. but only
>>> limited ssd space. We also have a 40GBe network and a decently fast
>>> file server.
>>> 
>>> My question is simple but I didnt find an answer anywhere: What are the
>>> best practices for the working directory on mesos-agent nodes? Should
>>> we keep the working directory local or is it reasonable to use a nfs
>>> mounted folder? We implemented both and they seem to work fine, but I
>>> would rather like to follow "best practices".
>>> 
>>> Thanks and cheers
>>> 
>>> Tom
>>> 
>> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Marathon] Weird restart behaviour

[Tomek Janiszewski]

Can you provide some logs? Why do you have to restart Mesos masters? What
happen to Mesos?

wt., 27 cze 2017 o 15:49 użytkownik Jean-Baptiste 
napisał:

> Hi there,
>
>
> I have new question for you guys. I’ve noticed that each time I have to
> restart Marathon, I have to restart Mesos masters services too.
>
>
> So in the order:
>
>
>1. Restart Marathon (fail by doesn’t allow to deploy anything)
>2. Restart Mesos masters
>3. Restart Marathon
>
>
> Then it works.
>
>
> Actually, I really have no idea of what is going on because I can’t see
> the link they could have together to generate this.
>
> Do you guys think that it could be related to *Debian* startup scripts?
> Or any idea?
>
>
> Thanks!
>
> *Misc:*
>
>- *Marathon version:** 1.4.5-1.0.654.debian81
>- *Mesos version:** 1.1.0-2.0.107.debian81
>- *Debian version:** 8.7
>
>
> --
>
> Jean-Baptiste FAREZ
>

[Marathon] Weird restart behaviour

[Jean-Baptiste]

Hi there,


I have new question for you guys. I’ve noticed that each time I have to
restart Marathon, I have to restart Mesos masters services too.


So in the order:


   1. Restart Marathon (fail by doesn’t allow to deploy anything)
   2. Restart Mesos masters
   3. Restart Marathon


Then it works.


Actually, I really have no idea of what is going on because I can’t see the
link they could have together to generate this.

Do you guys think that it could be related to *Debian* startup scripts? Or
any idea?


Thanks!

*Misc:*

   - *Marathon version:** 1.4.5-1.0.654.debian81
   - *Mesos version:** 1.1.0-2.0.107.debian81
   - *Debian version:** 8.7


-- 

Jean-Baptiste FAREZ

Ubuntu version for Apache Mesos Docker images

[Jörg Schad]

Dear community,
the current Mesos Docker image (provided by Mesosphere here:
https://hub.docker.com/r/mesosphere/mesos/) are currently based on Ubuntu
14.04.

Given that 14.04 is rather old, we would like to switch to 16.04 as a
default base image.

Any concerns or comments?
Kapil & Joerg