Re: Agent Working Directory Best Practices

[Steven Schlansker]

> On Jun 26, 2017, at 5:30 PM, James Peach <jor...@gmail.com> wrote:
> 
> 
>> On Jun 26, 2017, at 4:05 PM, Steven Schlansker <sschlans...@opentable.com> 
>> wrote:
>> 
>> 
>>> On Jun 25, 2017, at 11:24 PM, Benjamin Mahler <bmah...@apache.org> wrote:
>>> 
>>> As a data point, as far as I'm aware, most users are using a local work 
>>> directory, not an NFS mounted one. Would love to hear from anyone on the 
>>> list if they are doing this, and if there are any subtleties that should be 
>>> documented.
>> 
>> We don't run NFS in particular but we did originally use a SAN -- two 
>> observations:
>> 
>> NFS (historically, maybe it's better now, but doubtful...) has really bad 
>> failure modes.
>> Network failures can cause serious hangs both in user-space and 
>> kernel-space.  Such
>> hangs can be impossible to clear without rebooting the machine, and in some 
>> edge cases
>> can even make it difficult or impossible to reboot the machine via normal 
>> means.
> 
> You need to make sure to mount with the "intr" option.
> 
> https://speakerdeck.com/gnb/130-lca2008-nfs-tuning-secrets-d7

That's not without some caveats.  nfs(5):

The intr / nointr mount option is deprecated after kernel 2.6.25. Only SIGKILL 
can interrupt a pending NFS operation on these kernels, and if specified, this 
mount option is ignored to provide backwards compatibility with older kernels.
Using the intr option is preferred to using the soft option because it is 
significantly less likely to result in data corruption.

...

NB: A so-called "soft" timeout can cause silent data corruption in certain 
cases. As such, use the soft option only when client responsiveness is more 
important than data integrity. Using NFS over TCP or increasing the value of 
the retrans option may mitigate some of the risks of using the soft option.


So, 'intr' is deprecated / removed on any reasonable kernel, and 'soft' has 
silent data corruption issues.
Typical Linux, having a broken implementation that then points you to instead 
use a deprecated / removed implementation :)

I'm sure there's a way to get NFS working great.  Just pointing out that you'll 
need an expert to take ownership of it!

> 
>> 
>> Network attached drives (our SAN) are less reliable, slower, and more complex
>> (read: more failure modes) than local disk.  It's also a really big single 
>> point
>> of failure.  So far our only true cluster outages have been due to failure of
>> the SAN, since it took down all nodes at once -- once we removed the SAN, 
>> future
>> failures had islands of availability and any properly written application
>> could continue running (obviously without network resources) through the 
>> incident.
>> 
>> Maybe this isn't a huge deal for your use case, which might differ from ours.
>> For us, it was enough of a problem that we now purchase local SSD scratch 
>> space
>> for every node just so that we have some storage we can depend on a bit more
>> than network attached storage.
>> 
>>> 
>>> On Thu, Jun 22, 2017 at 11:13 PM, <thomas.kurm...@artorg.unibe.ch> wrote:
>>> Hi,
>>> 
>>> We have a couple of server nodes mainly used for computational tasks in
>>> our mesos cluster. These servers have beefy cpus, gpus etc. but only
>>> limited ssd space. We also have a 40GBe network and a decently fast
>>> file server.
>>> 
>>> My question is simple but I didnt find an answer anywhere: What are the
>>> best practices for the working directory on mesos-agent nodes? Should
>>> we keep the working directory local or is it reasonable to use a nfs
>>> mounted folder? We implemented both and they seem to work fine, but I
>>> would rather like to follow "best practices".
>>> 
>>> Thanks and cheers
>>> 
>>> Tom
>>> 
>> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: cron-like scheduling in mesos framework?

[Steven Schlansker]

Singularity also supports this well, https://github.com/HubSpot/Singularity 
(and it is a framework where you can do both "regular" and "scheduled" tasks 
through one interface)

> On Jan 6, 2017, at 8:47 AM, Tomas Barton  wrote:
> 
> Hi,
> 
> try Chronos framework https://mesos.github.io/chronos/
> 
> Tomas
> 
> On 6 January 2017 at 17:40, l vic  wrote:
> Hi,
> Is there a way to schedule mesos framework task for execution at certain 
> day/time?
> Thank youm
> -V
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Monitoring at container level

[Steven Schlansker]

We use Graphite and ran into similar problems with huge metric namespaces.
We use the Singularity framework which provides both the task "request id" 
(name)
and "instance number" (0..N) to the task.

So we set our Graphite namespace to be "request-number" e.g. "myservice-3"
This has the downside of discontinuous data when you deploy a new release
but we haven't had too many issues due to that in practice.


> On Jul 7, 2016, at 1:26 PM, Krish  wrote:
> 
> I have had a good experience so far with bosun and scollector with cadvisor.
> Check it out at bosun.org.
> 
> 
> On Friday 8 July 2016, Pradeep Chhetri  wrote:
> Hi Michal,
> 
> Do have a look at sysdig (http://www.sysdig.org). It is basically an 
> open-source tool which provides container insights. Maybe your will find 
> something helpful over there.
> 
> To tackle the case of new metrics for new containers, maybe you should tag 
> metrics by service-name instead of container id. (Graphite doesn't have 
> concept of tags but something like opentsdb and influxdb do have. I don't see 
> a reason to replace graphite for that. You can use your service-name (which 
> the container is representing) instead of hostname in the metrics name)
> 
> On Fri, Jul 8, 2016 at 1:18 AM, Michał Łowicki  wrote:
> Hi,
> 
> Before introducing Mesos we're using mainly Graphite / Grafana. Ideally we 
> would like to have metrics per container as an easy way to detect if problem 
> touches only single, subset of containers or it's global.
> 
> Unfortunately using Graphite for that is far from being perfect. Having 
> container identifier as a part of metric has many negative implications like 
> having tons of new metrics every release on Marathon (new containers = new 
> identifiers).
> 
> Investigated InfluxDB so far but project isn't mature enough as still 
> components like 
> https://github.com/influxdata/telegraf/blob/master/plugins/inputs/statsd/README.md#influx-statsd
>  have major blockers:
> 
> COMING SOON: there will be a way to specify multiple fields.
> 
> What do you use to monitor your Mesos clusters and f.ex. to detect that some 
> containers are having issues?
> 
> --
> BR,
> Michał Łowicki
> 
> 
> 
> --
> Regards,
> Pradeep Chhetri
> 
> 
> --
> 
> Thumb typed mail
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Consequences of health-check timeouts?

[Steven Schlansker]

> On May 18, 2016, at 10:44 AM, haosdent  wrote:
> 
> >In re executor_shutdown_grace_period: how would this enable the task 
> >(MongoDB) to terminate gracefully? (BTW: I am fairly certain that the mongo 
> >STDOUT as captured by Mesos shows that it received signal 15 just before it 
> >said good-bye). My naive understanding of this grace period is that it 
> >simply delays the termination of the executor.

I'm not 100% sure this is related or helpful, but be aware that we believe 
there is a bug in the Docker
containerizer's handling of logs during shutdown:

https://issues.apache.org/jira/browse/MESOS-5195

We spent a lot of time debugging why our application was not shutting down as 
we expected,
only to find that the real problem was that Mesos was losing all logs sent 
during
shutdown.

> 
> If you use DockerContainerizer, mesos use executor_shutdown_grace_period as 
> the shutdown gracefully timeout for task as well. If you use 
> MesosContainerizer, it would send SIGTERM(15) first. After 3 seconds, if the 
> task is still alive, Mesos would send SIGKILL(9) to the task again.
> 
> >I'm not sure what the java task is. This took place on the mesos-master node 
> >and none of our applications runs there. It runs master, Marathon, and ZK. 
> >Maybe the java task is Marathon or ZK?
> 
> Not sure about this, maybe others have similar experience on this, do 
> Marathon or Zookeeper abnormal at that time? Could you provide the log of 
> mesos-master/mesos-slave when accident happened as well?
> 
> 
> On Wed, May 18, 2016 at 7:11 PM, Paul Bell  wrote:
> Hi Hasodent,
> 
> Thanks for your reply.
> 
> In re executor_shutdown_grace_period: how would this enable the task 
> (MongoDB) to terminate gracefully? (BTW: I am fairly certain that the mongo 
> STDOUT as captured by Mesos shows that it received signal 15 just before it 
> said good-bye). My naive understanding of this grace period is that it simply 
> delays the termination of the executor.
> 
> The following snippet is rom /var/log/syslog. I believe it shows the stack 
> trace (largely in the kernel) that led to mesos-master being blocked for more 
> than 120 seconds. Please note that immediately above (before) the blocked 
> mesos-master is a blocked jbd2/dm. Immediately below (after) the blocked 
> mesos-master is a blocked java task. I'm not sure what the java task is. This 
> took place on the mesos-master node and none of our applications runs there. 
> It runs master, Marathon, and ZK. Maybe the java task is Marathon or ZK?
> 
> Thanks again.
> 
> -Paul
> May 16 20:06:53 71 kernel: [193339.890848] INFO: task mesos-master:4013 
> blocked for more than 120 seconds.
> 
> May 16 20:06:53 71 kernel: [193339.890873]   Not tainted 
> 3.13.0-32-generic #57-Ubuntu
> 
> May 16 20:06:53 71 kernel: [193339.890889] "echo 0 > 
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 
> May 16 20:06:53 71 kernel: [193339.890912] mesos-masterD 88013fd94440 
> 0  4013  1 0x
> 
> May 16 20:06:53 71 kernel: [193339.890914]  880137429a28 0002 
> 880135778000 880137429fd8
> 
> May 16 20:06:53 71 kernel: [193339.890916]  00014440 00014440 
> 880135778000 88013fd94cd8
> 
> May 16 20:06:53 71 kernel: [193339.890918]  88013ffd34b0 0002 
> 81284630 880137429aa0
> 
> May 16 20:06:53 71 kernel: [193339.890919] Call Trace:
> 
> May 16 20:06:53 71 kernel: [193339.890922]  [] ? 
> start_this_handle+0x590/0x590
> 
> May 16 20:06:53 71 kernel: [193339.890924]  [] 
> io_schedule+0x9d/0x140
> 
> May 16 20:06:53 71 kernel: [193339.890925]  [] 
> sleep_on_shadow_bh+0xe/0x20
> 
> May 16 20:06:53 71 kernel: [193339.890927]  [] 
> __wait_on_bit+0x62/0x90
> 
> May 16 20:06:53 71 kernel: [193339.890929]  [] ? 
> start_this_handle+0x590/0x590
> 
> May 16 20:06:53 71 kernel: [193339.890930]  [] 
> out_of_line_wait_on_bit+0x77/0x90
> 
> May 16 20:06:53 71 kernel: [193339.890932]  [] ? 
> autoremove_wake_function+0x40/0x40
> 
> May 16 20:06:53 71 kernel: [193339.890934]  [] ? 
> wake_up_bit+0x25/0x30
> 
> May 16 20:06:53 71 kernel: [193339.890936]  [] 
> do_get_write_access+0x2ad/0x4f0
> 
> May 16 20:06:53 71 kernel: [193339.890938]  [] ? 
> __getblk+0x2d/0x2e0
> 
> May 16 20:06:53 71 kernel: [193339.890939]  [] 
> jbd2_journal_get_write_access+0x27/0x40
> 
> May 16 20:06:53 71 kernel: [193339.890942]  [] 
> __ext4_journal_get_write_access+0x3b/0x80
> 
> May 16 20:06:53 71 kernel: [193339.890946]  [] 
> ext4_reserve_inode_write+0x70/0xa0
> 
> May 16 20:06:53 71 kernel: [193339.890948]  [] ? 
> ext4_dirty_inode+0x40/0x60
> 
> May 16 20:06:53 71 kernel: [193339.890949]  [] 
> ext4_mark_inode_dirty+0x44/0x1f0
> 
> May 16 20:06:53 71 kernel: [193339.890951]  [] 
> ext4_dirty_inode+0x40/0x60
> 
> May 16 20:06:53 71 kernel: [193339.890953]  [] 
> __mark_inode_dirty+0x10a/0x2d0
> 
> May 16 20:06:53 71 kernel: [193339.890956]  [] 
> 

Potential serious issue when upgrading OpenJDK8 to >= 8u77 with Debian / Ubuntu packaging

[Steven Schlansker]

Hello Mesos fans,

I just wanted to alert you to a potentially disastrous incompatibility
introduced in the last few OpenJDK packages released for the popular
"openjdk-r" Ubuntu PPA.

Per Debian bug 815475:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=815475

The OpenJDK packaging changed the virtual packages it provides.
Notably, it no longer provides "java-runtime-headless", which Mesos depends on:
http://bazaar.launchpad.net/%7Eopenjdk/openjdk/openjdk8/revision/672/debian/control

This means that if you use the Mesosphere .deb packaging and the OpenJDK
Ubuntu / Debian repos, when you upgrade from < 8u77 to >= 8u77, the Mesos
package may be uninstalled.  If you happen to run Puppet, it calls apt with
the "-f" force option by default, which will do all this without even
telling you...

I believe I have a fix here:
https://github.com/mesosphere/mesos-deb-packaging/pull/77

I would love critical review -- this surprised us and crippled our
production clusters, and I got this out in a panic, so I am not 100% sure
it is the right fix... hopefully this note saves someone else a really really
bad day.

Wishing package management a slow and painful death,
Steven

(PS... maybe this is good evidence that the Mesos project needs
more support around understanding packaging and distribution management)



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: mesos docker vs native container

[Steven Schlansker]

> On Apr 25, 2016, at 12:37 AM, vincent gromakowski 
>  wrote:
> 
> I am very interesting in getting some feedback of people who has moved from 
> native container through Docker specially from network performance 
> perspective.
> DCOS has been open sourced and I like all automation it brings with 
> frameworks but it seems everything is running in docker ?
> I am looking for the smack stack for which network perf is important.

If you use the Docker default bridge network stack, I have heard of performance 
impact
up to 30% (depending on workload / kernel version / etc)

We use "host" networking which moderately reduces security but
all but eliminates the performance impact on the network.

You should make sure to do tests that this is true in your environment
but I believe it is possible to get near native performance if you
are willing to test things thoroughly and report issues as they arise.



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Mesos-master url in HA

[Steven Schlansker]

I personally believe that this is not a sufficient workaround -- what
if the master is failing over, and your autoscaler happens to redirect
to a master which just lost leadership?

This solution is inherently racy and leads to the end user writing extra code
to work around it, and even then can still result in extremely difficult
to diagnose bugs.

I'd filed an issue on this a while ago (0.20 days):
https://issues.apache.org/jira/browse/MESOS-1865

but unfortunately it is still not resolved.

> On Apr 13, 2016, at 12:44 AM, Alexander Rojas  wrote:
> 
> Hi guillermo,
> 
> The master has the `/redirect` endpoint which should point you to the current 
> leader.
> 
>> On 13 Apr 2016, at 08:20, Guillermo Rodriguez  wrote:
>> 
>> Hi,
>> 
>> I have 3 mesos master setup for HA. One has the lead.
>> 
>> http://172.31.35.90:5050/master/slaves   <<< leader, returns slave list
>> http://172.31.35.91:5050/master/slaves   <<< follower, returns empty list
>> http://172.31.35.92:5050/master/slaves   <<< follower, returns empty list
>> 
>> I have an URL mesos-master.mydomain.com pointing to the leader and that 
>> works fine because it returns the slave list which I need for my autoscaler. 
>> But I'm afraid if the master fails the URL will no longer be valid. So I 
>> added the three IPs to the router (AWS Route53)  so it would round robin, 
>> but of course this will return an empty list sometimes because it hits a 
>> follower which returns empty.
>> 
>> So my question is, is it possible to redirect the call from the followers to 
>> the leader master?
>> 
>> Thanks.
>> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: What are the invalid-user.log files?

[Steven Schlansker]

We are seeing the same thing, using Mesosphere .debs on Ubuntu:

mesos-master.mesos1-qa-sf.invalid-user.log.WARNING.20160209-221625.12071
mesos-master.mesos1-qa-sf.invalid-user.log.WARNING.20160223-211310.1456
mesos-master.mesos1-qa-sf.invalid-user.log.WARNING.20160223-211857.5347

What if the fallback was to using getpwuid(getuid()) or the like instead of 
"invalid-user" ?


> On Mar 18, 2016, at 9:58 AM, Peter Steele  wrote:
> 
> In my case I am using the CentOS 7 set of rpms  from mesosphere...
> 
> On 03/18/2016 09:14 AM, Pradeep Chhetri wrote:
>> I installed mesos using mesosphere debian repository.
>> 
>> On Fri, Mar 18, 2016 at 4:10 PM, haosdent  wrote:
>> cool, you install mesos by deb or build it from source directly?
>> 
>> On Mar 19, 2016 12:06 AM, "Pradeep Chhetri"  
>> wrote:
>> I think it is an upstart issue. I am noticing this issue in Ubuntu 14.04 
>> which uses upstart. This is what i can debug so far:
>> 
>> 
>> According to this documentation:  
>> http://upstart.ubuntu.com/cookbook/#job-environment
>> 
>> When Upstart runs a job, it provides it with a very restrictive environment 
>> which contains just two system variables:
>> 
>>  • TERM
>>  • PATH
>> 
>> These are the exact two variables i can notice in proc filesystem set as 
>> well.
>> 
>> I will post if i find something more.
>> 
>> 
>> On Fri, Mar 18, 2016 at 2:54 PM, Peter Steele  wrote:
>> Yes, I see that as well. Also the case for the mesos master. That explains 
>> the invalid-user, but why isn't these processes picking up $USER?
>> 
>> 
>> On 03/18/2016 07:31 AM, Pradeep Chhetri wrote:
>>> I can see that USER environment variable is not set for mesos-slave process 
>>> from /proc//environ.
>>> 
>>> On Fri, Mar 18, 2016 at 2:19 PM, Pradeep Chhetri 
>>>  wrote:
>>> My mesos cluster also produces logs like 
>>> mesos-slave.ip-172-31-45-33.invalid-user.log.ERROR. I guess log file name 
>>> shouldnt affect you all because generally you ship these logs in some 
>>> centralized logging system like logstash/splunk and search there by tags
>>> 
>>> On Fri, Mar 18, 2016 at 1:46 PM, Peter Steele  wrote:
>>> What would you suggest to troubleshoot this? Clear something isn't quite 
>>> right if my log files are called "invalid-user". That said, I have managed 
>>> to get a containerized application up and running so whatever is wrong 
>>> isn't fatal. Just ugly...
>>> 
>>> 
>>> On 03/17/2016 10:42 AM, haosdent wrote:
 Not sure why glog could not get USER environment variable correctly after 
 looking its code. But should not affect you running mesos.
 
 On Fri, Mar 18, 2016 at 1:11 AM, haosdent  wrote:
 root should be fine.
 
 On Fri, Mar 18, 2016 at 12:53 AM, Peter Steele  wrote:
 My USER var is root. Do I need to create a non-root user for mesos to run 
 under?
 
 
 On 03/17/2016 09:22 AM, haosdent wrote:
> glog get the user name by environment variable "USER" in Linux. 
> https://github.com/google/glog/blob/master/src/utilities.cc#L290-L302 I 
> think you could check the environment variable "USER" before you start 
> Mesos slave.
> 
> On Fri, Mar 18, 2016 at 12:19 AM, haosdent  wrote:
> invalid-user is from glog 
> https://github.com/google/glog/blob/master/src/logging.cc#L1036
> 
> >W0317 06:26:41.178268   915 authenticator.cpp:511] No credentials 
> >provided, authentication requests will be refused
> >which I'm thinking might be related. What am I missing?
> 
> I think should not related. This is a warning message if you don't pass 
> `credentials` flag when start mesos master.
> 
> On Thu, Mar 17, 2016 at 10:07 PM, Peter Steele  wrote:
> I'm testing out mesos for the first time and after installing the 
> software I'm seeing numerous log files of the form
> 
> mesos-slave..invalid-user.log.INFO.20160317-062640.918
> 
> I don't see any errors in the logs themselves, but the fact that 
> "invalid-user" is part of the name makes me think I'm missing something 
> in my config. The mesos-master.WARNING log has the message
> 
> W0317 06:26:41.178268   915 authenticator.cpp:511] No credentials 
> provided, authentication requests will be refused
> 
> which I'm thinking might be related. What am I missing?
> 
> Peter
> 
> 
> 
> 
> --
> Best Regards,
> Haosdent Huang
> 
> 
> 
> --
> Best Regards,
> Haosdent Huang
 
 
 
 
 --
 Best Regards,
 Haosdent Huang
 
 
 
 --
 Best Regards,
 Haosdent Huang
>>> 
>>> 
>>> 
>>> 
>>> --
>>> Regards,
>>> Pradeep Chhetri
>>> 
>>> 
>>> 
>>> --
>>> Regards,
>>> Pradeep Chhetri
>> 
>> 
>> 
>> 
>> --
>> Regards,
>> 

Re: [VOTE] Release Apache Mesos 0.28.0 (rc1)

[Steven Schlansker]

> On Mar 3, 2016, at 5:43 PM, Vinod Kone  wrote:
> 
> Hi all,
> Please vote on releasing the following candidate as Apache Mesos 0.28.0.
> 0.28.0 includes the following:
> ...
>   * [MESOS-2840] - **Experimental** support for container images in Mesos
> containerizer (a.k.a. Unified Containerizer). This allows frameworks to
> launch Docker/Appc containers using Mesos containerizer without relying on
> docker daemon (engine) or rkt. The isolation of the containers is done 
> using
> isolators. Please refer to docs/container-image.md for currently supported
> features and limitations.

As of commit 46dcae5, there doesn't seem to be any such documentation?
I'm excited to try this feature out :)

https://github.com/apache/mesos/blob/master/docs/container-image.md (404)



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Mesos 0.27.1 (rc1)

[Steven Schlansker]

On Feb 16, 2016, at 4:52 PM, Michael Park  wrote:

> Hi all,
> 
> Please vote on releasing the following candidate as Apache Mesos 0.27.1.
> 

I filed a bug against 0.27.0 where Mesos can emit totally invalid JSON
in response to the /files/read.json endpoint:

https://issues.apache.org/jira/browse/MESOS-4642

I suppose it's too late at this point to get it considered for the 0.27.1 
release?
I would have pushed sooner but I didn't realize the next release would happen 
so quickly :)



> 
> 0.27.1 includes the following:
> 
> * Improved `systemd` integration.
> * Ability to disable `systemd` integration.
> 
> * Additional performance improvements to /state endpoint.
> * Removed duplicate "active" keys from the /state endpoint.
> 
> The CHANGELOG for the release is available at:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.27.1-rc1
> 
> 
> The candidate for Mesos 0.27.1 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/0.27.1-rc1/mesos-0.27.1.tar.gz
> 
> The tag to be voted on is 0.27.1-rc1:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.27.1-rc1
> 
> The MD5 checksum of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/0.27.1-rc1/mesos-0.27.1.tar.gz.md5
> 
> The signature of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/0.27.1-rc1/mesos-0.27.1.tar.gz.asc
> 
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
> 
> The JAR is up in Maven in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1102
> 
> Please vote on releasing this package as Apache Mesos 0.27.1!
> 
> The vote is open until Fri Feb 19 17:00:00 PST 2016 and passes if a majority 
> of at least 3 +1 PMC votes are cast.
> 
> [ ] +1 Release this package as Apache Mesos 0.27.1
> [ ] -1 Do not release this package because ...
> 
> Thanks,
> 
> Joris, MPark

Re: GLOG settings

[Steven Schlansker]

On Dec 10, 2015, at 11:22 AM, Vinod Kone  wrote:

> 
> On Thu, Dec 10, 2015 at 11:12 AM, Zameer Manji  wrote:
> the native library logs to stderr directly.
> 
> By default the library logs to stderr. But you can set Mesos/GLOG env 
> variables (e.g., MESOS_LOG_DIR) to make it write to a file instead.

If you really want to redirect it through Java logging, you could probably set 
up
a pipe in /tmp, write logs to that -- and in the Java program read it and 
re-log it.

Re: API client libraries

[Steven Schlansker]

As a Mesos user who wants to be more of a contributor but hates C++, I could 
volunteer to help work with the Java reference implementation.

I totally understand wanting to keep the various client libraries out of the 
Mesos release cycle and somewhat independent.  Maybe a model where the clients 
are not "officially" part of the Mesos project, but are instead simply blessed 
as being the reference implementations, makes the most sense to start with.  We 
can always merge them in later if a compelling reason shows up.

On Sep 2, 2015, at 12:01 PM, Artem Harutyunyan  wrote:

> Thanks for bringing this up, Vinod!
> 
> We have to make sure that there are reference library implementations for at 
> least Python, Java, and Go. They may end up being owned and maintained by the 
> community, but I feel that Mesos developers should at least kickstart the 
> process and incubate those libraries. Once the initial implementations of 
> those libraries are in place we should also make sure to have reference usage 
> examples for them (like we do right now with Rendler).
> 
> In any case, this is a very important topic so I will go ahead and add it to 
> tomorrow's community sync agenda.
> 
> Cheers,
> Artem.
> 
> On Wed, Sep 2, 2015 at 11:49 AM, Vinod Kone  wrote:
> Hi folks,
> 
> Now that the v1 scheduler HTTP API (beta) is on the verge of being
> released, I wanted to open up the discussion about client libraries for the
> API. Mainly around support and home for the libs.
> 
> One idea is that, going forward, the only supported client library would be
> C++ library which will live in the mesos repo. All other client libraries
> (java, python, go etc) will not be officially supported but linked to from
> our webpage/docs.
> 
> Pros:
> --> The PMC/committers won't have the burden to maintain client libraries
> in languages we don't have expertise in.
> --> Gives more control (reviews, releases) and attribution (could live in
> the author's org's or personal repo) to 3rd party client library authors
> 
> Cons:
> --> Might be a step backward because we would be officially dropping
> support for Java and Python. This is probably a good thing?
> --> No quality control of the libraries by the PMC. Need co-ordination with
> library authors to incorporate API changes. Could lead to bad user
> experience.
> 
> I've taken a quick look at what other major projects do and it looks like
> most of them officially support a few api libs and then link to 3rdparty
> libs.
> 
> Docker
> :
> No official library? Links to 3rd party libs.
> 
> GitHub : Official support for
> Ruby, .Net, Obj-C. Links to 3rd party libs.
> 
> Google : All
> official libraries? No links to 3rd party libs?
> 
> K8S : Official
> support for Go. Links to 3rd party libs.
> 
> Twitter : Official
> support for Java. Links to 3rd party libs.
> 
> 
> Is this the way we want to go? This does mean we won't need a mesos/commons
> repo because the project would be not be officially supporting 3rd party
> libs. The supported C++ libs will live in the mesos repo.
> 
> Thoughts?
> 

Re: mesos-slave crashing with CHECK_SOME

[Steven Schlansker]

I 100% agree with your philosophy here, and I suspect it's something shared in 
the Mesos community.

I just think that we can restrict the domain of the failure to a smaller 
reasonable window -- once you are in the context of "I am doing work to launch 
a specific task", there is a well defined "success / failure / here is an error 
message" path defined already.  Users expect tasks to fail and can see the 
errors.

I think that a lot of these assertions are in fact more appropriate as task 
failures.  But I agree that they should be fatal to *some* part of the system, 
just not the whole agent entirely.

On Sep 1, 2015, at 4:33 PM, Marco Massenzio <ma...@mesosphere.io> wrote:

> That's one of those areas for discussions that is so likely to generate a 
> flame war that I'm hesitant to wade in :)
> 
> In general, I would agree with the sentiment expressed there:
> 
> > If the task fails, that is unfortunate, but not the end of the world. Other 
> > tasks should not be affected.
> 
> which is, in fact, to large extent exactly what Mesos does; the example given 
> in MESOS-2684, as it happens, is for a "disk full failure" - carrying on as 
> if nothing had happened, is only likely to lead to further (and worse) 
> disappointment.
> 
> The general philosophy back at Google (and which certainly informs the design 
> of Borg[0]) was "fail early, fail hard" so that either (a) the service is 
> restarted and hopefully the root cause cleared or (b) someone (who can 
> hopefully do something) will be alerted about it.
> 
> I think it's ultimately a matter of scale: up to a few tens of servers, you 
> can assume there is some sort of 'log-monitor' that looks out for errors and 
> other anomalies and alerts humans that will then take a look and possibly 
> apply some corrective action - when you're up to hundreds or thousands 
> (definitely Mesos territory) that's not practical: the system should either 
> self-heal or crash-and-restart.
> 
> All this to say, that it's difficult to come up with a general *automated* 
> approach to unequivocally decide if a failure is "fatal" or could just be 
> safely "ignored" (after appropriate error logging) - in general, when in 
> doubt it's probably safer to "noisily crash & restart" and rely on the 
> overall system's HA architecture to take care of replication and consistency.
> (and an intelligent monitoring system that only alerts when some failure 
> threshold is exceeded).
> 
> From what I've seen so far (granted, still a novice here) it seems that Mesos 
> subscribes to this notion, assuming that Agent Nodes will come and go, and 
> usually Tasks survive (for a certain amount of time anyway) a Slave restart 
> (obviously, if the physical h/w is the ultimate cause of failure, well, then 
> all bets are off).
> 
> Having said all that - if there are areas where we have been over-eager with 
> our CHECKs, we should definitely revisit that and make it more 
> crash-resistant, absolutely.
> 
> [0] http://research.google.com/pubs/pub43438.html
> 
> Marco Massenzio
> Distributed Systems Engineer
> http://codetrips.com
> 
> On Mon, Aug 31, 2015 at 12:47 PM, Steven Schlansker 
> <sschlans...@opentable.com> wrote:
> 
> 
> On Aug 31, 2015, at 11:54 AM, Scott Rankin <sran...@motus.com> wrote:
> >
> > tag=mesos-slave[12858]:  F0831 09:37:29.838184 12898 slave.cpp:3354] 
> > CHECK_SOME(os::touch(path)): Failed to open file: No such file or directory
> 
> I reported a similar bug a while back:
> 
> https://issues.apache.org/jira/browse/MESOS-2684
> 
> This seems to be a class of bugs where some filesystem operations which may 
> fail for unforeseen reasons are written as assertions which crash the 
> process, rather than failing only the task and communicating back the error 
> reason.
> 
> 
> 

Re: mesos-slave crashing with CHECK_SOME

[Steven Schlansker]

On Aug 31, 2015, at 11:54 AM, Scott Rankin  wrote:
> 
> tag=mesos-slave[12858]:  F0831 09:37:29.838184 12898 slave.cpp:3354] 
> CHECK_SOME(os::touch(path)): Failed to open file: No such file or directory 

I reported a similar bug a while back:

https://issues.apache.org/jira/browse/MESOS-2684

This seems to be a class of bugs where some filesystem operations which may 
fail for unforeseen reasons are written as assertions which crash the process, 
rather than failing only the task and communicating back the error reason.

Re: Mesos Modifying User Group

[Steven Schlansker]

On Aug 12, 2015, at 3:28 PM, Nastooh Avessta (navesta) nave...@cisco.com 
wrote:

 Having a bit of a strange problem with Mesos 0.22, running Spark 1.4.0, on 
 Docker 1.6 slaves. Part of my Spark program calls on a script that accesses a 
 GPU. I am able to run this script:
 1.   As Bash
 2.   Via Marathon
 3.   As part of a Spark program running as a standalone master
 However, when I try to run the same Spark program with Mesos as master, i.e., 
 spark-submit --master mesos://\`cat /etc/mesos/zk\` --deploy-mode client…, I 
 am not able to access dri devices, e.g., mfx init: /dev/dri/renderD128 fd 
 open failed. What seems to be happening is that the group membership of the 
 default user, in this case “ubuntu” is modified by Mesos, i.e., whereas under 
 cases 1-3, above, I get:
  
 $ id
 uid=1000(ubuntu) gid=1000(ubuntu) 
 groups=1000(ubuntu),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),102(netdev),999(docker)
 In case of Mesos, I get:
 uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),0(root)
  
 I am wondering if there are configuration parameters that can be passed to 
 Mesos to prevent it from modifying user groups?

Assuming your diagnosis here is correct,  this is actually a serious security 
issue -- notice how the group 0(root) was added!

Re: Get List of Active Slaves

[Steven Schlansker]

Unfortunately that sort of solution is also prone to races.
I do not think this is really possible (at least not even remotely elegantly) 
to solve externally to Mesos itself.

On Aug 4, 2015, at 4:49 PM, James DeFelice james.defel...@gmail.com wrote:

 If you're using mesos-dns I think you can query slave.mesos to get an a 
 record for each. I believe it responds to srv requests too.
 
 On Aug 4, 2015 7:29 PM, Steven Schlansker sschlans...@opentable.com wrote:
 Unfortunately this is racey.  If you redirect to a master just as it is 
 removed from leadership, you can still get bogus data, with no indication 
 anything went wrong.  Some people are reporting that this breaks tools that 
 generate HTTP proxy configurations.
 
 I filed this issue a while ago as 
 https://issues.apache.org/jira/browse/MESOS-1865
 
 On Aug 4, 2015, at 3:49 PM, Vinod Kone vinodk...@gmail.com wrote:
 
  Not today, no.
 
  But, you could either hit the /redirect endpoint on any master that 
  should redirect you to the leading master.
 
  On Tue, Aug 4, 2015 at 3:29 PM, Nastooh Avessta (navesta) 
  nave...@cisco.com wrote:
  I see. Nope, and pointing to the leading master shows the proper resultJ 
  Thanks.
 
  Is there a REST equivalent to mesos-resolve, so that one can ascertain who 
  is the leader without having to point  to the leader?
 
  Cheers,
 
 
 
  image001.jpg
 
  Nastooh Avessta
  ENGINEER.SOFTWARE ENGINEERING
  nave...@cisco.com
  Phone: +1 604 647 1527
 
  Cisco Systems Limited
  595 Burrard Street, Suite 2123 Three Bentall Centre, PO Box 49121
  VANCOUVER
  BRITISH COLUMBIA
  V7X 1J1
  CA
  Cisco.com
 
 
 
  image002.gifThink before you print.
 
  This email may contain confidential and privileged material for the sole 
  use of the intended recipient. Any review, use, distribution or disclosure 
  by others is strictly prohibited. If you are not the intended recipient (or 
  authorized to receive for the recipient), please contact the sender by 
  reply email and delete all copies of this message.
 
  For corporate legal information go to:
  http://www.cisco.com/web/about/doing_business/legal/cri/index.html
 
  Cisco Systems Canada Co, 181 Bay St., Suite 3400, Toronto, ON, Canada, M5J 
  2T3. Phone: 416-306-7000; Fax: 416-306-7099. Preferences - Unsubscribe – 
  Privacy
 
 
 
  From: Vinod Kone [mailto:vinodk...@gmail.com]
  Sent: Tuesday, August 04, 2015 3:19 PM
  To: user@mesos.apache.org
  Subject: Re: Get List of Active Slaves
 
 
 
  Is that the leading master?
 
 
 
  On Tue, Aug 4, 2015 at 3:09 PM, Nastooh Avessta (navesta) 
  nave...@cisco.com wrote:
 
  Hi
 
  Trying to get the list of active slaves, via cli, e.g. curl 
  http://10.4.50.80:5050/master/slaves | python -m json.tool and am not 
  getting the expected results.  The returned value is empty:
 
  {
 
  slaves: []
 
  }
 
  , whereas, looking at web gui I can see that there are deployed slaves. Am 
  I missing something?
 
  Cheers,
 
 
 
  image001.jpg
 
  Nastooh Avessta
  ENGINEER.SOFTWARE ENGINEERING
  nave...@cisco.com
  Phone: +1 604 647 1527
 
  Cisco Systems Limited
  595 Burrard Street, Suite 2123 Three Bentall Centre, PO Box 49121
  VANCOUVER
  BRITISH COLUMBIA
  V7X 1J1
  CA
  Cisco.com
 
 
 
  image002.gifThink before you print.
 
  This email may contain confidential and privileged material for the sole 
  use of the intended recipient. Any review, use, distribution or disclosure 
  by others is strictly prohibited. If you are not the intended recipient (or 
  authorized to receive for the recipient), please contact the sender by 
  reply email and delete all copies of this message.
 
  For corporate legal information go to:
  http://www.cisco.com/web/about/doing_business/legal/cri/index.html
 
  Cisco Systems Canada Co, 181 Bay St., Suite 3400, Toronto, ON, Canada, M5J 
  2T3. Phone: 416-306-7000; Fax: 416-306-7099. Preferences - Unsubscribe – 
  Privacy
 
 
 
 
 
 
 

Re: Get List of Active Slaves

[Steven Schlansker]

Unfortunately this is racey.  If you redirect to a master just as it is removed 
from leadership, you can still get bogus data, with no indication anything went 
wrong.  Some people are reporting that this breaks tools that generate HTTP 
proxy configurations.

I filed this issue a while ago as 
https://issues.apache.org/jira/browse/MESOS-1865

On Aug 4, 2015, at 3:49 PM, Vinod Kone vinodk...@gmail.com wrote:

 Not today, no.
 
 But, you could either hit the /redirect endpoint on any master that should 
 redirect you to the leading master.
 
 On Tue, Aug 4, 2015 at 3:29 PM, Nastooh Avessta (navesta) nave...@cisco.com 
 wrote:
 I see. Nope, and pointing to the leading master shows the proper resultJ 
 Thanks.
 
 Is there a REST equivalent to mesos-resolve, so that one can ascertain who is 
 the leader without having to point  to the leader?
 
 Cheers,
 
  
 
 image001.jpg
 
 Nastooh Avessta
 ENGINEER.SOFTWARE ENGINEERING
 nave...@cisco.com
 Phone: +1 604 647 1527
 
 Cisco Systems Limited
 595 Burrard Street, Suite 2123 Three Bentall Centre, PO Box 49121
 VANCOUVER
 BRITISH COLUMBIA
 V7X 1J1
 CA
 Cisco.com
 
  
 
 image002.gifThink before you print.
 
 This email may contain confidential and privileged material for the sole use 
 of the intended recipient. Any review, use, distribution or disclosure by 
 others is strictly prohibited. If you are not the intended recipient (or 
 authorized to receive for the recipient), please contact the sender by reply 
 email and delete all copies of this message.
 
 For corporate legal information go to:
 http://www.cisco.com/web/about/doing_business/legal/cri/index.html
 
 Cisco Systems Canada Co, 181 Bay St., Suite 3400, Toronto, ON, Canada, M5J 
 2T3. Phone: 416-306-7000; Fax: 416-306-7099. Preferences - Unsubscribe – 
 Privacy
 
  
 
 From: Vinod Kone [mailto:vinodk...@gmail.com] 
 Sent: Tuesday, August 04, 2015 3:19 PM
 To: user@mesos.apache.org
 Subject: Re: Get List of Active Slaves
 
  
 
 Is that the leading master?
 
  
 
 On Tue, Aug 4, 2015 at 3:09 PM, Nastooh Avessta (navesta) nave...@cisco.com 
 wrote:
 
 Hi
 
 Trying to get the list of active slaves, via cli, e.g. curl 
 http://10.4.50.80:5050/master/slaves | python -m json.tool and am not getting 
 the expected results.  The returned value is empty:
 
 {
 
 slaves: []
 
 }
 
 , whereas, looking at web gui I can see that there are deployed slaves. Am I 
 missing something?
 
 Cheers,
 
  
 
 image001.jpg
 
 Nastooh Avessta
 ENGINEER.SOFTWARE ENGINEERING
 nave...@cisco.com
 Phone: +1 604 647 1527
 
 Cisco Systems Limited
 595 Burrard Street, Suite 2123 Three Bentall Centre, PO Box 49121
 VANCOUVER
 BRITISH COLUMBIA
 V7X 1J1
 CA
 Cisco.com
 
  
 
 image002.gifThink before you print.
 
 This email may contain confidential and privileged material for the sole use 
 of the intended recipient. Any review, use, distribution or disclosure by 
 others is strictly prohibited. If you are not the intended recipient (or 
 authorized to receive for the recipient), please contact the sender by reply 
 email and delete all copies of this message.
 
 For corporate legal information go to:
 http://www.cisco.com/web/about/doing_business/legal/cri/index.html
 
 Cisco Systems Canada Co, 181 Bay St., Suite 3400, Toronto, ON, Canada, M5J 
 2T3. Phone: 416-306-7000; Fax: 416-306-7099. Preferences - Unsubscribe – 
 Privacy
 
  
 
  
 
 

Re: Debugging framework registration from inside docker

[Steven Schlansker]

On Jun 10, 2015, at 10:10 AM, James Vanns jvanns@gmail.com wrote:

 Hi. When attempting to run my scheduler inside a docker container in 
 --net=bridge mode it never receives acknowledgement or a reply to that 
 request. However, it works fine in --net=host mode. It does not listen on any 
 port as a service so does not expose any.
 
 The scheduler receives the mesos master (leader) from zookeeper fine but 
 fails to register the framework with that master. It just loops trying to do 
 so - the master sees the registration but deactivates it immediately as 
 apparently it disconnects. It doesn't disconnect but is obviously 
 unreachable. I see the reason for this in the sendto() and the master log 
 file -- because the internal docker bridge IP is included in the POST and 
 perhaps that is how the master is trying to talk back
 to the requesting framework?? 
 
 Inside the container is this;
 tcp0  0 0.0.0.0:44431   0.0.0.0:*   LISTEN
   1/scheduler
 
 This is not my code! I'm at a loss where to go from here. Anyone got any 
 further suggestions
 to fix this?

You may need to try setting LIBPROCESS_IP and LIBPROCESS_PORT to hide the fact 
that you are on a virtual Docker interface.

Re: [DISCUSS] Renaming Mesos Slave

[Steven Schlansker]

On Jun 8, 2015, at 1:12 AM, Aaron Carey aca...@ilm.com wrote:

 I've been following this thread with interest, it draws a lot of parallels 
 with similar problems my wife faces as a teacher (and I imagine this happens 
 in other government/public sector organisations, earlier in this thread James 
 pointed me to an interested Wikipedia article which suggested this also 
 happens occasionally in software: eg County of Los Angeles in 2003). Every 
 few years teachers are told to change the words used to describe various 
 things related to kids with minority backgrounds, from underprivileged 
 families or with disabilities and so on, usually to stop other children from 
 using them as derogatory terms or insults. It works for a while and then the 
 pupils catch on and start using the new words and the cycle repeats.
 
 I guess the point I'm trying to make here is that if you do decide to change 
 the naming of master/slave because some naughty programmers in the community 
 have been using the terms offensively, you better make damn sure you choose 
 new terms which aren't likely to cause offence in the future and require the 
 whole renaming process to run again. Which is why I'm voting for:
 
 +1 Gru/Minion

Which then is great right up until Universal Pictures sues the Apache 
foundation to get Gru changed.  Plus master/slave is immediately obvious to 
anyone working in software.  I had to search the web to even figure out what 
Gru was, and then it was not even the first result... ( 
http://en.wikipedia.org/wiki/Main_Intelligence_Directorate_%28Russia%29 )

 
 There could also be another option: These terms are all being used to 
 describe a master/slave relationship, the mesos master is in charge, it 
 assigns work to the slaves and ensures that they carry it out. I'd suggest 
 that whatever you call this pair, the relationship will always be one of 
 domination and servitude. Perhaps what is really needed here is to get rid of 
 the concept of a master altogether and re-architect mesos so all nodes in the 
 cluster are equal and reach a consensus together about work distribution and 
 so on?

I propose all processes, regardless of function, should be mesos-comrade to 
ensure none of them feel slighted :)

 
 
 From: Nikolay Borodachev [nbo...@adobe.com]
 Sent: 06 June 2015 04:34
 To: user@mesos.apache.org
 Subject: RE: : [DISCUSS] Renaming Mesos Slave
 
 +1 master/slave ?C no need to change
  
 From: Sam Salisbury [mailto:samsalisb...@gmail.com] 
 Sent: Friday, June 05, 2015 8:31 AM
 To: user@mesos.apache.org
 Subject: Re: : [DISCUSS] Renaming Mesos Slave
  
 Master/Minion +1
  
 On 5 June 2015 at 15:14, CCAAT cc...@tampabay.rr.com wrote:
 
 +1 master/slave, no change needed.  is the same as
 master/slaveI.E. keep the nomenclature as it currently is
 
 This means keep the name 'master' and keep the name 'slave'.
 
 
 Are you applying fuzzy math or kalman filters to your summations below?
 
 It looks to me, tallying things up, Master is kept as it is
 and 'Slave' is kept as it is. There did not seem to be any consensus
 on the new names if the pair names are updated. Or you can vote separately on 
 each name? On an  real ballot, you enter the choices,
 vote according to your needs, tally the results and publish them.
 Applying a 'fuzzy filter' to what has occurred in this debate so far
 is ridiculous.
 
 Why not repost the question like this or something on a more fair
 voting preference:
 
 
 Please vote for your favourite Name-pair in Mesos, for what is currently
 Master-Slave. Note Master-Slave is the no change vote option.
 
 [] Master-Slave
 [] Mesos-Slave
 [] Mesos-Minion
 [] Master-Minion
 [] Master-Follower
 [] Mesos-Follower
 [] Master-worker
 [] Mesos-worker
 [] etc etc
 
 -
 
 
 Tally the result and go from there.
 James
 
 
 
 
 On 06/05/2015 04:27 AM, Adam Bordelon wrote:
 Wow, what a response! Allow me to attempt to summarize the sentiment so far.
 
 Let's start with the implicit question,
 _0. Should we rename Mesos Slave?_
 +1 (Explicit approval) 12, including 7 from JIRA
 +0.5 (Implicit approval, suggested alternate name) 18
 -0.5 (Some disapproval, wouldn't block it) 5, including 1 from JIRA
 -1 (Strong disapproval) 16
 
 _1. What should we call the Mesos Slave node/host/machine?_
 Worker: +10, -2
 Agent: +6
 Follower (+Leader): +4, -1
 Minion: +2, -1
 Drone (+Director/Queen): +2
 Resource-Agent/Provider: +2
 
 _2. What should we call the mesos-slave process (could be the same)?_
 Pretty much everybody says that it should be the same as the node.
 
 _3. Do we need to rename Mesos Master too?_
 Most say No, except when slave's new name has a preferred pairing (e.g.
 Follower/Leader)
 
 _4. How will we phase in the new name and phase out the old name?_
 To calm any fears, we would have to go through a full deprecation cycle,
 introducing the new name in one release, while maintaining
 symlinks/aliases/duplicate-endpoints for the old name. In 

Re: Accounting for Mesos resource usage

[Steven Schlansker]

This is all great!  One thing that surprises is that it integrates
with the Mesos master.

Were I coding it, I'd see it as something that naturally should live
as a separate application outside - IMO everything that can be outside
of the Mesos master should be, especially given that:

* Developing in C++ sucks (/me ducks)
* Anything that goes wrong with it brings down the Mesos master

But if it integrates naturally, obviously we'd use it :)

Do let us know when you file that issue

On Dec 18, 2014, at 12:05 PM, Niklas Nielsen n...@qni.dk wrote:

 Hi Steven,
 
 Alex Rukletsov and I worked on this as a proof-of-concept piece in the 
 mesos-master last week, providing the same kind of graphs as you describe in 
 the dashboard.
 We have a good idea about how to implement this now and we can start a 
 discussion on JIRA on how to proceed (I can create it shortly).
 My first thought is that this should be pluggable; having something similar 
 to status update decorators
 Alongside hanging key-value pairs of the status update, you can keep track of 
 the life-time/size of tasks and do the resource math.
 
 There are some interesting problems to solve when it gets to master 
 fail-over, but let's try to enumerate those in the ticket.
 
 Thanks,
 Niklas
 
 On Thu, Dec 18, 2014 at 11:56 AM, Steven Schlansker 
 sschlans...@opentable.com wrote:
 I am running a corporate Mesos cluster, shared by a number of teams and 
 projects.
 We are looking to get some insight into our usage of precious computing 
 resources.  For example, I'd like to be able to present a report breaking 
 down CPU-hour and RAM GB-hour utilization by service, team, or other relevant 
 grouping.
 
 How I'd imagine this works:
 
 * Collect Mesos statistics per task (allocated CPU, CPU utilization, 
 allocated memory, memory utilization, disk utilization) periodically (say, 
 once a minute)
 * Collect task metadata from a pluggable source (mapping from Mesos task to 
 service name, team name, any other metadata you wish to use to group tasks)
 * Generate dashboard / reports by aggregating task data over axes provided by 
 metadata input
 
 Has anyone started on such a project?
 
 Thanks,
 Steven
 
 
 
 -- 
 Niklas

Re: Accounting for Mesos resource usage

[Steven Schlansker]

It'd be nice to build the solution in a way that is independent of Singularity 
- it seems like
the vast majority of the data can be collected from Mesos with then only a 
small piece that integrates
with Singularity or Marathon or Chronos or whatnot.  Then all the different 
framework users could contribute
to the shared accounting system.


On Dec 18, 2014, at 12:48 PM, Thomas Petr tp...@hubspot.com wrote:

 We (HubSpot) currently have a cron job that enumerates all tasks running on 
 the slaves and pushes resource usage data into OpenTSDB. We then use lead.js 
 to query / visualize this data. The cron job isn't open source, but I could 
 look into releasing it if anyone is interested. I've also thought about 
 adding this functionality into our Singularity framework, but if it was 
 directly supported by the mesos master (pumping task resource usage into 
 graphite / OpenTSDB), that'd be pretty cool.
 
 -Tom
 
 On Thu, Dec 18, 2014 at 3:25 PM, Andrew Ortman 
 and...@impossibleventures.com wrote:
 I imagine we will also be running into the same need. Our plan right now was 
 to write an quick service that polls the API the dashboard uses to retrieve 
 metric information for each slave and then pipe that data directly to 
 something like graphite for logging. I haven’t looked too much into this yet 
 though
 
 
 On Dec 18, 2014, at 2:05 PM, Niklas Nielsen n...@qni.dk wrote:
 
 Hi Steven,
 
 Alex Rukletsov and I worked on this as a proof-of-concept piece in the 
 mesos-master last week, providing the same kind of graphs as you describe in 
 the dashboard.
 We have a good idea about how to implement this now and we can start a 
 discussion on JIRA on how to proceed (I can create it shortly).
 My first thought is that this should be pluggable; having something similar 
 to status update decorators
 Alongside hanging key-value pairs of the status update, you can keep track 
 of the life-time/size of tasks and do the resource math.
 
 There are some interesting problems to solve when it gets to master 
 fail-over, but let's try to enumerate those in the ticket.
 
 Thanks,
 Niklas
 
 On Thu, Dec 18, 2014 at 11:56 AM, Steven Schlansker 
 sschlans...@opentable.com wrote:
 I am running a corporate Mesos cluster, shared by a number of teams and 
 projects.
 We are looking to get some insight into our usage of precious computing 
 resources.  For example, I'd like to be able to present a report breaking 
 down CPU-hour and RAM GB-hour utilization by service, team, or other 
 relevant grouping.
 
 How I'd imagine this works:
 
 * Collect Mesos statistics per task (allocated CPU, CPU utilization, 
 allocated memory, memory utilization, disk utilization) periodically (say, 
 once a minute)
 * Collect task metadata from a pluggable source (mapping from Mesos task to 
 service name, team name, any other metadata you wish to use to group tasks)
 * Generate dashboard / reports by aggregating task data over axes provided 
 by metadata input
 
 Has anyone started on such a project?
 
 Thanks,
 Steven
 
 
 
 -- 
 Niklas
 

Re: Accounting for Mesos resource usage

[Steven Schlansker]

This is great, and will help a lot!  I do wish there was a way to store 
structured data rather than just a string, but this will do to start :)

On Dec 18, 2014, at 2:22 PM, Benjamin Mahler benjamin.mah...@gmail.com wrote:

 Hey Steven,
 
 We introduced a field on the ExecutorInfo called source [1] (which we 
 didn't document well, sorry about that). This field should also be introduced 
 on the TaskInfo [2] for those who just use the built-in CommandExecutor.
 
 When launching an executor or a task, you can specify a source that allows 
 you to identify the executor for monitoring / reporting purposes. This is how 
 we do our container monitoring at Twitter:
 
 (1) Frameworks set source to a meaningful string: e.g. 
 prod.ads-team.ads-server.0. (environment, team, service, instance).
 (2) An internal system collects minutely data from /monitor/statistics.json 
 on the slave.
 (3) The metrics include all those you listed above (disk utilization is 
 coming). Once these are stored in an internal TSDB, dashboards, queries, and 
 reports can be constructed for the relevant services, teams, etc.
 
 I hope this helps, IMHO this is probably the simplest path forward for you, 
 as you don't need to have any pluggable functionality built-in to mesos to 
 get what you need.
 
 Ben
 
 [1] https://github.com/apache/mesos/blob/0.21.0/include/mesos/mesos.proto#L283
 [2] https://issues.apache.org/jira/browse/MESOS-486
 
 On Thu, Dec 18, 2014 at 11:56 AM, Steven Schlansker 
 sschlans...@opentable.com wrote:
 I am running a corporate Mesos cluster, shared by a number of teams and 
 projects.
 We are looking to get some insight into our usage of precious computing 
 resources.  For example, I'd like to be able to present a report breaking 
 down CPU-hour and RAM GB-hour utilization by service, team, or other relevant 
 grouping.
 
 How I'd imagine this works:
 
 * Collect Mesos statistics per task (allocated CPU, CPU utilization, 
 allocated memory, memory utilization, disk utilization) periodically (say, 
 once a minute)
 * Collect task metadata from a pluggable source (mapping from Mesos task to 
 service name, team name, any other metadata you wish to use to group tasks)
 * Generate dashboard / reports by aggregating task data over axes provided by 
 metadata input
 
 Has anyone started on such a project?
 
 Thanks,
 Steven
 

Resize Mesos master quorum

[Steven Schlansker]

I'm looking to be able to increase (3 - 5) and decrease (5 - 3) the number of 
Mesos masters I run.
There doesn't seem to be any documentation on this procedure on the website.

What is the correct way to approach this?

Thanks,
Steven

Accounting for Mesos resource usage

[Steven Schlansker]

I am running a corporate Mesos cluster, shared by a number of teams and 
projects.
We are looking to get some insight into our usage of precious computing 
resources.  For example, I'd like to be able to present a report breaking down 
CPU-hour and RAM GB-hour utilization by service, team, or other relevant 
grouping.

How I'd imagine this works:

* Collect Mesos statistics per task (allocated CPU, CPU utilization, allocated 
memory, memory utilization, disk utilization) periodically (say, once a minute)
* Collect task metadata from a pluggable source (mapping from Mesos task to 
service name, team name, any other metadata you wish to use to group tasks)
* Generate dashboard / reports by aggregating task data over axes provided by 
metadata input

Has anyone started on such a project?

Thanks,
Steven

Re: Monitoring Mesos slave/master processes

[Steven Schlansker]

On Dec 9, 2014, at 3:45 PM, Gary Malouf malouf.g...@gmail.com wrote:

 We did this in the past with Nagios, but I was wondering if there was a 
 recommended way from others using in production.

I wrote a Nagios plugin for it

https://github.com/opentable/nagios-mesos

Re: Mesos inside Docker

[Steven Schlansker]

On Dec 2, 2014, at 8:22 AM, Jeremy Jongsma jer...@barchart.com wrote:

 What is the current state of running Mesos inside Docker? I ran across a 
 thread that indicates it may be working now, but I have not seen any Docker 
 images for Mesos from authoritative sources.
 
 https://www.mail-archive.com/user@mesos.apache.org/msg01616.html
 
 Is anybody doing this in production? Are there known issues?
 
 We already have a base AMI for deploying and updating Docker containers 
 across our infrastructure, so adding Mesos inside that would be a simple way 
 to start using it.

I'm also interested in this, and can help contribute to an existing Mesos 
packaging if we find one or can collaborate on creating one if there's no 
suitable project already.

Re: Rocket

[Steven Schlansker]

On Dec 1, 2014, at 11:22 AM, Niklas Nielsen nik...@mesosphere.io wrote:

 Huge +1
 
 On 1 December 2014 at 11:10, Tim Chen t...@mesosphere.io wrote:
 Hi all,
 
 Per the announcement from CoreOS about Rocket 
 (https://coreos.com/blog/rocket/) , it seems to be an exciting containerizer 
 runtime that has composable isolation/components, better security and image 
 specification/distribution.
 
 All of these design goals also fits very well into Mesos, where in Mesos we 
 also have a pluggable isolators model and have been experiencing some pain 
 points with our existing containerizers around image distribution and 
 security as well.
 
 I'd like to propose to integrate Rocket into Mesos with a new Rocket 
 containerizer, where I can see we can potentially integrate our existing 
 isolators into Rocket runtime.
 
 Like to learn what you all think,

We too are using Docker and find the monolithic nature to be obnoxious at 
times.  If Rocket were easy to run side-by-side I'd certainly give it a whirl 
and switch to it if it works at least as well.

Re: Multiple disks with Mesos

[Steven Schlansker]

On Oct 7, 2014, at 4:06 PM, Arunabha Ghosh arunabha...@gmail.com wrote:

 Hi,
  I would like to run Mesos slaves on machines that have multiple disks. 
 According to the Mesos configuration page I can specify a work_dir argument 
 to the slaves. 
 
 1) Can the work_dir argument contain multiple directories ?
 
 2) Is the work_dir where Mesos will place all of its data ? So If I started a 
 task on Mesos, would the slave place the task's data (stderr, stdout, task 
 created directories) inside work_dir ?

We stitch our disks together before Mesos gets its hands on it using a 
technology such as LVM or btrfs, so that the work_dir is spread across the 
multiple disks transparently.

Re: Mesos 0.20.0 with Docker registry availability

[Steven Schlansker]

Would it be possible to have a mode where it tries to pull, but then does not 
fail solely due to the fail of a pull?  In particular, we use tags to indicate 
which build should be deployed e.g. “foo-server:production” tag vs 
“foo-server:staging” tags.

On Sep 4, 2014, at 11:05 PM, Tim Chen t...@mesosphere.io wrote:

 Hi Maxime,
 
 It is a very valid concern and that's why I've added a patch that should go 
 out in 0.20.1 to not do a docker pull on every run anymore.
 
 Mesos will still try to docker pull when the image isn't available locally 
 (via docker inspect), but only once.
 
 The downside ofcourse is that you're not able to automatically get the latest 
 tagged image, but I think it's worth while price to may to gain the benefits 
 of not depending on registry, able to run local images and more.
 
 Tim
 
 
 On Thu, Sep 4, 2014 at 10:50 PM, Maxime Brugidou maxime.brugi...@gmail.com 
 wrote:
 Hi,
 
 The current Docker integration in 0.20 does a docker pull from the registry 
 before running any task. This means that your entire Mesos cluster becomes 
 unusable if the registry goes down.
 
 The docs allow you to configure a custom .dockercfg for your tasks to point 
 to a private docker registry.
 
 However it is not easy to run an HA docker registry. The docker-registry 
 project recommend using S3 storage buy this is definitely not an option for 
 some people.
 
 I know that for regular artifacts, Mesos can use HDFS storage and you can run 
 your HDFS datanodes as Mesos tasks.
 
 So even if I attempt to have a docker registry storage in HDFS (which is not 
 supported by docker-registry at the moment), I am stuck on a chicken and egg 
 problem. I want to have as little services outside of Mesos as possible and 
 it is hard to maintain HA services (especially outside of Mesos).
 
 Is there anyone running Mesos with Docker in production without S3? I am 
 trying to make all the services outside of Mesos (the infra services that 
 are necessary to run Mesos like DNS, Haproxy, Chef server... etc) either HA 
 or not critical for the cluster to run. The docker registry is a new piece of 
 infra outside of Mesos that is critical...
 
 Best,
 Maxime
 
 

Re: Trying out Docker containerizer: fails with no interesting output

[Steven Schlansker]

Thanks, that was definitely a gotcha!

On Sep 4, 2014, at 4:27 PM, David Greenberg dsg123456...@gmail.com wrote:

 Even though command is blank, you must set shell to false. There's a ticket 
 for this that I don't have off-hand. 
 
 On Thursday, September 4, 2014, Steven Schlansker sschlans...@opentable.com 
 wrote:
 I am trying to integrate the Docker containerizer into the Singularity 
 framework (https://github.com/HubSpot/Singularity)
 
 I have filled in my containerInfo:
   containerInfo: {
 type: DOCKER,
 docker: {
   image: registry.mesos-vpcqa.otenv.com/demo-server:latest-master
 }
   }
 
 I am leaving ‘command’ blank as I wish to have the default command executed.
 
 Interestingly enough, the Docker containerizer seems to pass on the TaskInfo:
 I0904 23:12:13.941849 32497 exec.cpp:132] Version: 0.20.0
 I0904 23:12:13.947185 32500 exec.cpp:206] Executor registered on slave 
 20140826-180432-1594050058-5050-23467-6
 F0904 23:12:13.949040 32502 executor.cpp:134] Check failed: 
 task.command().has_value() Shell command of task 
 DemoService-DemoService.18-1409872328807-2-10-us_west_2a is not specified!
 *** Check failure stack trace: ***
 @ 0x7f54dc76b4bd  google::LogMessage::Fail()
 @ 0x7f54dc76d2fd  google::LogMessage::SendToLog()
 @ 0x7f54dc76b0ac  google::LogMessage::Flush()
 @ 0x7f54dc76dbf9  google::LogMessageFatal::~LogMessageFatal()
 @   0x42abc2  
 mesos::internal::CommandExecutorProcess::launchTask()
 @ 0x7f54dc7160e2  process::ProcessManager::resume()
 @ 0x7f54dc7163ec  process::schedule()
 @ 0x7f54db190182  start_thread
 @ 0x7f54daebd38d  (unknown)
 Aborted
 
 but if I add some (otherwise unnecessary) URIs to fetch it then is happy to 
 accept it
 
   uris: [ https://my.s3.amazonaws.com/something.tar.gz”]
 
 then I start to see Docker containerizer output in the slave log:
 
 I0904 22:58:24.092098  1288 docker.cpp:815] Starting container 
 '479dedef-4312-4441-be72-6570caaf2b8a' for task 
 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' (and executor 
 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a') of framework 
 'Singularity'
 I0904 22:58:28.327430  1288 slave.cpp:2535] Monitoring executor 
 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' of framework 
 'Singularity' in container '479dedef-4312-4441-be72-6570caaf2b8a'
 I0904 22:58:28.368155  1291 status_update_manager.cpp:320] Received status 
 update TASK_RUNNING (UUID: a428c917-973d-4c7e-a8a3-43fdf9b49e99) for task 
 DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework 
 Singularity
 I0904 22:58:29.331042  1291 slave.cpp:2087] Handling status update 
 TASK_FINISHED (UUID: de9b8003-9b61-4b87-98fc-32b01f875a95) for task 
 DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework 
 Singularity from executor(1)@10.70.6.186:56465
 I0904 22:58:30.689954  1291 docker.cpp:1720] Executor for container 
 '479dedef-4312-4441-be72-6570caaf2b8a' has exited
 I0904 22:58:30.690076  1291 docker.cpp:1545] Destroying container 
 '479dedef-4312-4441-be72-6570caaf2b8a'
 I0904 22:58:30.690500  1291 docker.cpp:1642] Running docker kill on container 
 '479dedef-4312-4441-be72-6570caaf2b8a'
 
 stderr:
 WARNING: Logging before InitGoogleLogging() is written to STDERR
 I0904 22:58:24.317865 26387 fetcher.cpp:126] Downloading ‘xxx' to 
 '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx'
 I0904 22:58:24.387418 26387 fetcher.cpp:64] Extracted resource 
 '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx'
  into 
 '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a'
 I0904 22:58:27.870132 26811 exec.cpp:132] Version: 0.20.0
 I0904 22:58:28.155580 26819 exec.cpp:206] Executor registered on slave 
 20140826-180432-1594050058-5050-23467-8
 
 stdout:
 
 Registered executor on 10.70.6.186
 Starting task DemoService-DemoService.15-1409871503777-2-10-us_west_2a
 Forked command at 26949
 /bin/sh -c exit `docker wait mesos-479dedef-4312-4441-be72-6570caaf2b8a`
 Command exited with status 0 (pid: 26949)
 
 When I try to lookup the Docker container with ‘docker ps -a’ it looks like 
 no container was ever created, and then the ‘docker wait’ exits immediately.
 
 Any clues as to how I could debug such a situation?
 Thanks!
 Steven

Trying out Docker containerizer: fails with no interesting output

[Steven Schlansker]

I am trying to integrate the Docker containerizer into the Singularity 
framework (https://github.com/HubSpot/Singularity)

I have filled in my containerInfo:
  containerInfo: {
type: DOCKER,
docker: {
  image: registry.mesos-vpcqa.otenv.com/demo-server:latest-master
}
  }

I am leaving ‘command’ blank as I wish to have the default command executed.

Interestingly enough, the Docker containerizer seems to pass on the TaskInfo:
I0904 23:12:13.941849 32497 exec.cpp:132] Version: 0.20.0
I0904 23:12:13.947185 32500 exec.cpp:206] Executor registered on slave 
20140826-180432-1594050058-5050-23467-6
F0904 23:12:13.949040 32502 executor.cpp:134] Check failed: 
task.command().has_value() Shell command of task 
DemoService-DemoService.18-1409872328807-2-10-us_west_2a is not specified!
*** Check failure stack trace: ***
@ 0x7f54dc76b4bd  google::LogMessage::Fail()
@ 0x7f54dc76d2fd  google::LogMessage::SendToLog()
@ 0x7f54dc76b0ac  google::LogMessage::Flush()
@ 0x7f54dc76dbf9  google::LogMessageFatal::~LogMessageFatal()
@   0x42abc2  mesos::internal::CommandExecutorProcess::launchTask()
@ 0x7f54dc7160e2  process::ProcessManager::resume()
@ 0x7f54dc7163ec  process::schedule()
@ 0x7f54db190182  start_thread
@ 0x7f54daebd38d  (unknown)
Aborted

but if I add some (otherwise unnecessary) URIs to fetch it then is happy to 
accept it

  uris: [ https://my.s3.amazonaws.com/something.tar.gz”]

then I start to see Docker containerizer output in the slave log:

I0904 22:58:24.092098  1288 docker.cpp:815] Starting container 
'479dedef-4312-4441-be72-6570caaf2b8a' for task 
'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' (and executor 
'DemoService-DemoService.15-1409871503777-2-10-us_west_2a') of framework 
'Singularity'
I0904 22:58:28.327430  1288 slave.cpp:2535] Monitoring executor 
'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' of framework 
'Singularity' in container '479dedef-4312-4441-be72-6570caaf2b8a'
I0904 22:58:28.368155  1291 status_update_manager.cpp:320] Received status 
update TASK_RUNNING (UUID: a428c917-973d-4c7e-a8a3-43fdf9b49e99) for task 
DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework 
Singularity
I0904 22:58:29.331042  1291 slave.cpp:2087] Handling status update 
TASK_FINISHED (UUID: de9b8003-9b61-4b87-98fc-32b01f875a95) for task 
DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework 
Singularity from executor(1)@10.70.6.186:56465
I0904 22:58:30.689954  1291 docker.cpp:1720] Executor for container 
'479dedef-4312-4441-be72-6570caaf2b8a' has exited
I0904 22:58:30.690076  1291 docker.cpp:1545] Destroying container 
'479dedef-4312-4441-be72-6570caaf2b8a'
I0904 22:58:30.690500  1291 docker.cpp:1642] Running docker kill on container 
'479dedef-4312-4441-be72-6570caaf2b8a'
 
stderr:
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0904 22:58:24.317865 26387 fetcher.cpp:126] Downloading ‘xxx' to 
'/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx'
I0904 22:58:24.387418 26387 fetcher.cpp:64] Extracted resource 
'/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx'
 into 
'/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a'
I0904 22:58:27.870132 26811 exec.cpp:132] Version: 0.20.0
I0904 22:58:28.155580 26819 exec.cpp:206] Executor registered on slave 
20140826-180432-1594050058-5050-23467-8
 
stdout:
 
Registered executor on 10.70.6.186
Starting task DemoService-DemoService.15-1409871503777-2-10-us_west_2a
Forked command at 26949
/bin/sh -c exit `docker wait mesos-479dedef-4312-4441-be72-6570caaf2b8a` 
Command exited with status 0 (pid: 26949)

When I try to lookup the Docker container with ‘docker ps -a’ it looks like no 
container was ever created, and then the ‘docker wait’ exits immediately.

Any clues as to how I could debug such a situation?
Thanks!
Steven