Re: Deprecating /stats.json endpoints
This change has landed in head: commit d9ba9199a8c8357ab13a1b14f8ee63409c5ac310 (HEAD, origin/master, origin/HEAD, master) Author: Dominic Hamon dha...@twitter.com Commit: Dominic Hamon dha...@twitter.com Deprecate /stats.json endpoint and supporting code. Review: https://reviews.apache.org/r/30296 On Mon, Jan 26, 2015 at 4:23 PM, Dominic Hamon dha...@twopensource.com wrote: An addendum: This will also deprecate the task status counts from /state.json as they are already present in the new endpoint. /state.json will still retain the current state of every task. On Mon, Jan 26, 2015 at 2:31 PM, Dominic Hamon dha...@twopensource.com wrote: For a few releases now, we've had parity on master and slave between the old metrics endpoints (/stats.json) and the new endpoint (/metrics/snapshot). As per MESOS-2058 https://issues.apache.org/jira/browse/MESOS-2058, the old endpoints will be deprecated in 0.23. 0.22 will be released with a note to this effect. Please update any endpoint scraping tools. -- Dominic Hamon | @mrdo | Twitter *There are no bad ideas; only good ideas that go horribly wrong.* -- Dominic Hamon | @mrdo | Twitter *There are no bad ideas; only good ideas that go horribly wrong.* -- Dominic Hamon | @mrdo | Twitter *There are no bad ideas; only good ideas that go horribly wrong.*
Deprecating /stats.json endpoints
For a few releases now, we've had parity on master and slave between the old metrics endpoints (/stats.json) and the new endpoint (/metrics/snapshot). As per MESOS-2058 https://issues.apache.org/jira/browse/MESOS-2058, the old endpoints will be deprecated in 0.23. 0.22 will be released with a note to this effect. Please update any endpoint scraping tools. -- Dominic Hamon | @mrdo | Twitter *There are no bad ideas; only good ideas that go horribly wrong.*
Re: Rocket
Instead of considering the Rocket runtime as implemented, we should instead consider how we can implement their specification. A community is always healthier when there are multiple implementations of a specification, and through implementing it we may find ways to improve it. Also, this allows us to be a strong voice in the community and provide value through a C++ implementation. I've created a JIRA ticket https://issues.apache.org/jira/browse/MESOS-2162 to track any thoughts on this. On Mon, Dec 1, 2014 at 11:10 AM, Tim Chen t...@mesosphere.io wrote: Hi all, Per the announcement from CoreOS about Rocket ( https://coreos.com/blog/rocket/) , it seems to be an exciting containerizer runtime that has composable isolation/components, better security and image specification/distribution. All of these design goals also fits very well into Mesos, where in Mesos we also have a pluggable isolators model and have been experiencing some pain points with our existing containerizers around image distribution and security as well. I'd like to propose to integrate Rocket into Mesos with a new Rocket containerizer, where I can see we can potentially integrate our existing isolators into Rocket runtime. Like to learn what you all think, Thanks! -- Dominic Hamon | @mrdo | Twitter *There are no bad ideas; only good ideas that go horribly wrong.*
Re: Programmatic Mesos version detection/extraction?
There is a /state.json endpoint you can find that contains a 'version' key. On Wed, Nov 12, 2014 at 1:39 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, Is there a way to detect which version of Mesos one is running? Is there an API for that, or a constant with this value, or maybe an MBean or some other way to get to this info? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ -- Dominic Hamon | @mrdo | Twitter *There are no bad ideas; only good ideas that go horribly wrong.*
Re: Programmatic Mesos version detection/extraction?
/stats.json is deprecated /state.json is not sorry for the confusing names :) On Wed, Nov 12, 2014 at 2:52 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Thanks Dominic! I believe I read that /state.json is deprecated, right? Is the version info available via /metrics/status, too? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Wed, Nov 12, 2014 at 4:44 PM, Dominic Hamon dha...@twitter.com wrote: There is a /state.json endpoint you can find that contains a 'version' key. On Wed, Nov 12, 2014 at 1:39 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, Is there a way to detect which version of Mesos one is running? Is there an API for that, or a constant with this value, or maybe an MBean or some other way to get to this info? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ -- Dominic Hamon | @mrdo | Twitter *There are no bad ideas; only good ideas that go horribly wrong.* -- Dominic Hamon | @mrdo | Twitter *There are no bad ideas; only good ideas that go horribly wrong.*
Re: Why rely on url scheme for fetching?
. Hadoop/HDFS in general is pretty flexible when it comes to being able to fetch stuff from urls and has the ability to fetch a large number of types of urls and can be extended by adding configuration into the conf/hdfs-site.xml and core-site.xml What I am proposing is that we refactor the fetcher.cpp to prefer to use the hdfs (using hdfs/hdfs.hpp) to do all the fetching if HADOOP_HOME is set and $HADOOP_HOME/bin/hadoop is available. This logic already exists and we can just use it. The fallback logic for using net::download or local file copy is may be left in place for installations that do not have hadoop configured. This means that if hadoop is present we can directly fetch urls such as tachyon://... snackfs:// ... cfs:// ftp://... s3://... http:// ... file:// with no extra effort. This makes up for a much better experience when it comes to debugging and extensibility. What do others think about this? - Ankur -- Dominic Hamon | @mrdo | Twitter *There are no bad ideas; only good ideas that go horribly wrong.*
Re: build failure on mac os x
libsubversion was added as a dependency recently. However it's lack should have been caught by the configure step. I don't know enough about osx and libsubversion to know if there's something special you need to do. On Nov 1, 2014 2:05 AM, Ankur Chauhan an...@malloc64.com wrote: I am trying to build the latest (master) mesos source and keep getting this error: In file included from ../../src/state/log.cpp:25:0: ../../3rdparty/libprocess/3rdparty/stout/include/stout/svn.hpp:21:23: fatal error: svn_delta.h: No such file or directory #include svn_delta.h ^ compilation terminated. make[2]: *** [state/libstate_la-log.lo] Error 1 make[2]: *** Waiting for unfinished jobs mv -f log/.deps/liblog_la-recover.Tpo log/.deps/liblog_la-recover.Plo mv -f state/.deps/libstate_la-in_memory.Tpo state/.deps/libstate_la-in_memory.Plo libtool: compile: g++-4.9 -DPACKAGE_NAME=\mesos\ -DPACKAGE_TARNAME=\mesos\ -DPACKAGE_VERSION=\0.21.0\ -DPACKAGE_STRING=\mesos 0.21.0\ -DPACKAGE_BUGREPORT=\\ -DPACKAGE_URL=\\ -DPACKAGE=\mesos\ -DVERSION=\0.21.0\ -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\.libs/\ -DHAVE_PTHREAD=1 -DHAVE_LIBZ=1 -DHAVE_LIBCURL=1 -DHAVE_LIBAPR_1=1 -DHAVE_LIBSVN_SUBR_1=1 -DHAVE_LIBSVN_DELTA_1=1 -DHAVE_LIBSASL2=1 -DMESOS_HAS_JAVA=1 -I. -I../../src -Wall -Werror -DLIBDIR=\/usr/local/lib\ -DPKGLIBEXECDIR=\/usr/local/libexec/mesos\ -DPKGDATADIR=\/usr/local/share/mesos\ -I../../include -I../../3rdparty/libprocess/include -I../../3rdparty/libprocess/3rdparty/stout/include -I../include -I../include/mesos -I../3rdparty/libprocess/3rdparty/boost-1.53.0 -I../3rdparty/libprocess/3rdparty/picojson-4f93734 -I../3rdparty/libprocess/3rdparty/protobuf-2.5.0/src -I../3rdparty/libprocess/3rdparty/glog-0.3.3/src -I../3rdparty/libprocess/3rdparty/glog-0.3.3/src -I../3rdparty/leveldb/include -I../3rdparty/zookeeper-3.4.5/src/c/include -I../3rdparty/zookeeper-3.4.5/src/c/generated -I../3rdparty/libprocess/3rdparty/protobuf-2.5.0/src -I/usr/include/subversion-1 -I/usr/include/apr-1 -I/usr/include/apr-1.0 -D_THREAD_SAFE -g1 -O0 -Wno-unused-local-typedefs -std=c++11 -DGTEST_USE_OWN_TR1_TUPLE=1 -MT state/libstate_la-leveldb.lo -MD -MP -MF state/.deps/libstate_la-leveldb.Tpo -c ../../src/state/leveldb.cpp -o state/libstate_la-leveldb.o /dev/null 21 mv -f state/.deps/libstate_la-leveldb.Tpo state/.deps/libstate_la-leveldb.Plo make[1]: *** [all] Error 2 make: *** [all-recursive] Error 1 Any idea why this is happening? -- Ankur
Re: 'make check' triggered an oom situation
I don't know if anyone has successfully built mesos in a VM on EC2. Linking specifically takes a lot of memory. I don't have specific numbers though, I'm afraid. On Mon, Oct 6, 2014 at 3:30 PM, Valeriu Mutu vale...@kanjoya.com wrote: Hello, I'm new to Mesos and have successfully installed it and begun playing with it. When running 'make check' I ran into an out-of-memory situation and the vm was rebooted: Oct 6 22:00:08 ip-10-239-168-141 kernel: Task in /mesos_test/bc077378-d71b-499f-adea-2d7c933e5db6 killed as a result of limit of /mesos_test/bc077378-d71b-499f-adea-2d7c933e5db6 [...] Oct 6 22:00:08 ip-10-239-168-141 kernel: Memory cgroup out of memory: Kill process 3120 (balloon-executo) score 993 or sacrifice child Oct 6 22:00:08 ip-10-239-168-141 kernel: Killed process 3120, UID 0, (balloon-executo) total-vm:861460kB, anon-rss:97384kB, file-rss:2768kB What are the minimum hardware requirements for running 'make check'? I'm using an m3.xlarge ec2 instance. -- Valeriu Mutu -- Dominic Hamon | @mrdo | Twitter *There are no bad ideas; only good ideas that go horribly wrong.*
Re: Mesos language bindings in the wild
. -- Counterpoint - Spark (curating) libraries Spark bundles 1st order interface libraries as part of a curated core. You are guaranteed that the core will inter-operate, and PySpark is given 1st class standing. -- This is a bad idea, unless there is a plan to hedge the risk. -Tim - Original Message - From: yifan myan...@msn.com http://JAVASCRIPT-BLOCKED; To: user@mesos.apache.org http://JAVASCRIPT-BLOCKED; Sent: Monday, July 14, 2014 7:10:34 PM Subject: Re: Mesos language bindings in the wild Hi Tim, I found that in zookeeper, they also separate the bindings from the core. https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings So, IMHO, I think it should be the maintainer's responsibility to keep the binding in healthy state, with clear documentation of which version of the mesos core they supports. Yifan On 07/14/2014 11:30 AM, Tim St Clair wrote: So I fear the fragmentation that can occur if we provide native bindings outside of the core, unless there is some mechanism for testing, a well established versioning scheme. IMHO, priority inversion on 'versioning' should come before bindings to ensure we adhere to policy. Thoughts? -Tim - Original Message - From: Tom Arnfeld t...@duedil.com http://JAVASCRIPT-BLOCKED; To: d...@mesos.apache.org http://JAVASCRIPT-BLOCKED; Cc: user@mesos.apache.org http://JAVASCRIPT-BLOCKED; Sent: Friday, July 11, 2014 10:22:59 AM Subject: Re: Mesos language bindings in the wild Very exciting. I'd vote +1 for splitting them out. Especially if you look at the common way of using Go imports, just stick the project on GitHub and import it directly using github.com/mesos/mesos-go or similar. I guess one argument is that you have more fragmentation of the code (e.g every library has it's own copy of the protos) but I'm not sure that's a bad thing. Just my two cents. Looking forward to this! On 11 Jul 2014, at 16:59, Thomas Rampelberg tho...@saunter.org http://JAVASCRIPT-BLOCKED; wrote: I've started preparing the python bindings to hopefully take this route ( https://reviews.apache.org/r/23224/ would love some reviews! ). In fact, there is already a native python implementation of both libprocess and the framework apis! ( https://github.com/wickman/pesos/ , https://github.com/wickman/compactor ). What are the benefits of bindings being part of the project source itself instead of having blessed implementations like mesos-python where the source and versioning becomes separate? I've been running into difficulties making automake and python's build tools play nicely together. It seems like there'd be more flexibility in general by splitting them out. On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen nik...@mesosphere.io http://JAVASCRIPT-BLOCKED; wrote: I just wanted to clarify - native, meaning _no_ dependency to libmesos and native to its language (only Go, only Python and so on) i.e. use the low-level API. Sorry for the confusion, Niklas On 10 July 2014 15:55, Dominic Hamon dha...@twopensource.com http://JAVASCRIPT-BLOCKED; wrote: In my dream world, we wouldn't need any native bindings. I can imagine having example frameworks or starter frameworks that use the low-level API (the wire protocol with protocol buffers for message passing), but nothing like we have that needs C or JNI, etc. On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen nik...@mesosphere.io http://JAVASCRIPT-BLOCKED; wrote: Hi all, I wanted to start a discussion around the language bindings in the wild (Go, Haskell, native Python, Go, Java and so on) and possibly get to a strategy where we start bringing those into Mesos proper. As most things points towards, it will probably make sense to focus on the native bindings leveraging the low-level API. To name one candidate to start with, we are especially interested in getting Go native support in Mesos proper (and in a solid state). So Vladimir, we'd be super thrilled to start collaborating with you on your current work. We are interested to hear what thoughts you all might have on this. Thanks, Niklas -- Gu Yifan -- Cheers, Timothy St. Clair Red Hat Inc. -- Cheers, Timothy St. Clair Red Hat Inc. -- Vladimir Vivien
Re: Mesos language bindings in the wild
I'm a fan of splitting these things out as you end up with various implementations, some of which will be more suited for some applications that others. It also encourages exploration of the API in various ways. As an aside, I've been playing with a go version too at https://github.com/dominichamon/gozer. It's not meant to be anything but an API test-case. On Fri, Jul 11, 2014 at 8:22 AM, Tom Arnfeld t...@duedil.com wrote: Very exciting. I'd vote +1 for splitting them out. Especially if you look at the common way of using Go imports, just stick the project on GitHub and import it directly using github.com/mesos/mesos-go or similar. I guess one argument is that you have more fragmentation of the code (e.g every library has it's own copy of the protos) but I'm not sure that's a bad thing. Just my two cents. Looking forward to this! On 11 Jul 2014, at 16:59, Thomas Rampelberg tho...@saunter.org wrote: I've started preparing the python bindings to hopefully take this route ( https://reviews.apache.org/r/23224/ would love some reviews! ). In fact, there is already a native python implementation of both libprocess and the framework apis! (https://github.com/wickman/pesos/ , https://github.com/wickman/compactor ). What are the benefits of bindings being part of the project source itself instead of having blessed implementations like mesos-python where the source and versioning becomes separate? I've been running into difficulties making automake and python's build tools play nicely together. It seems like there'd be more flexibility in general by splitting them out. On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen nik...@mesosphere.io wrote: I just wanted to clarify - native, meaning _no_ dependency to libmesos and native to its language (only Go, only Python and so on) i.e. use the low-level API. Sorry for the confusion, Niklas On 10 July 2014 15:55, Dominic Hamon dha...@twopensource.com wrote: In my dream world, we wouldn't need any native bindings. I can imagine having example frameworks or starter frameworks that use the low-level API (the wire protocol with protocol buffers for message passing), but nothing like we have that needs C or JNI, etc. On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen nik...@mesosphere.io wrote: Hi all, I wanted to start a discussion around the language bindings in the wild (Go, Haskell, native Python, Go, Java and so on) and possibly get to a strategy where we start bringing those into Mesos proper. As most things points towards, it will probably make sense to focus on the native bindings leveraging the low-level API. To name one candidate to start with, we are especially interested in getting Go native support in Mesos proper (and in a solid state). So Vladimir, we'd be super thrilled to start collaborating with you on your current work. We are interested to hear what thoughts you all might have on this. Thanks, Niklas -- Dominic Hamon | @mrdo | Twitter *There are no bad ideas; only good ideas that go horribly wrong.*
Re: Mesos language bindings in the wild
I'm a fan of splitting these things out as you end up with various implementations, some of which will be more suited for some applications that others. It also encourages exploration of the API in various ways. As an aside, I've been playing with a go version too at https://github.com/dominichamon/gozer. It's not meant to be anything but an API test-case. On Fri, Jul 11, 2014 at 8:22 AM, Tom Arnfeld t...@duedil.com wrote: Very exciting. I'd vote +1 for splitting them out. Especially if you look at the common way of using Go imports, just stick the project on GitHub and import it directly using github.com/mesos/mesos-go or similar. I guess one argument is that you have more fragmentation of the code (e.g every library has it's own copy of the protos) but I'm not sure that's a bad thing. Just my two cents. Looking forward to this! On 11 Jul 2014, at 16:59, Thomas Rampelberg tho...@saunter.org wrote: I've started preparing the python bindings to hopefully take this route ( https://reviews.apache.org/r/23224/ would love some reviews! ). In fact, there is already a native python implementation of both libprocess and the framework apis! (https://github.com/wickman/pesos/ , https://github.com/wickman/compactor ). What are the benefits of bindings being part of the project source itself instead of having blessed implementations like mesos-python where the source and versioning becomes separate? I've been running into difficulties making automake and python's build tools play nicely together. It seems like there'd be more flexibility in general by splitting them out. On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen nik...@mesosphere.io wrote: I just wanted to clarify - native, meaning _no_ dependency to libmesos and native to its language (only Go, only Python and so on) i.e. use the low-level API. Sorry for the confusion, Niklas On 10 July 2014 15:55, Dominic Hamon dha...@twopensource.com wrote: In my dream world, we wouldn't need any native bindings. I can imagine having example frameworks or starter frameworks that use the low-level API (the wire protocol with protocol buffers for message passing), but nothing like we have that needs C or JNI, etc. On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen nik...@mesosphere.io wrote: Hi all, I wanted to start a discussion around the language bindings in the wild (Go, Haskell, native Python, Go, Java and so on) and possibly get to a strategy where we start bringing those into Mesos proper. As most things points towards, it will probably make sense to focus on the native bindings leveraging the low-level API. To name one candidate to start with, we are especially interested in getting Go native support in Mesos proper (and in a solid state). So Vladimir, we'd be super thrilled to start collaborating with you on your current work. We are interested to hear what thoughts you all might have on this. Thanks, Niklas
Re: Mesos 0.19.0 stats.json endpoint Collectd plugin
Hi Dave The JIRA is here: https://issues.apache.org/jira/browse/MESOS-1036 There's a link in the comments to the design document. On Mon, Jul 7, 2014 at 11:14 AM, Dave Lester daveles...@gmail.com wrote: Hey Dominic, is there a specific review or JIRA issue that you could point to re: the new stats endpoint? Thanks, Dave On Thu, Jun 26, 2014 at 1:26 PM, Dominic Hamon dha...@twitter.com wrote: In 0.20, stats.json is completely represented by the new endpoint. stats.json will be deprecated in one of the next few versions. On Jun 26, 2014 1:10 PM, Ray Rodriguez rayrod2...@gmail.com wrote: Hey Ben, It looks like /metrics/snapshot is a subset of the metrics exposed from /stats.json. Is stats.json eventually going to be deprecated? Why collect less metrics in this case? On Thu, Jun 26, 2014 at 3:38 PM, Ray Rodriguez rayrod2...@gmail.com wrote: Yup totally makes sense. I'll update the endpoint. Thanks. On Thu, Jun 26, 2014 at 3:33 PM, Benjamin Mahler benjamin.mah...@gmail.com wrote: This is great! If only supporting 0.19.0+ I would recommend just collecting from /metrics/snapshot because it obviates /stats.json. On Thu, Jun 26, 2014 at 12:13 PM, Ray Rodriguez rayrod2...@gmail.com wrote: Hey everyone just thought I'd post a simple collectd python plugin that I just released which turns all stats returned from the Mesos stats.json endpoint into Collectd metrics. This plugin requires Mesos 0.19.0+ and collectd 4.9+ Big shout out to phobos182(https://github.com/phobos182) who wrote the original Elasticsearch collectd python plugin upon which my plugin borrows from heavily. Mesos Collectd Python plugin: https://github.com/rayrod2030/collectd-mesos Hope this is useful to others and I welcome pull requests, comments and suggestions. Ray Rodriguez Twitter: @rayray2030 Github: rayrod2030
Re: 0.19.1
Hi Can you give some background as to why this is a critical fix? We try to minimise what we include in bug fix releases to avoid feature creep. Thanks On Jul 4, 2014 12:31 PM, Tom Arnfeld t...@duedil.com wrote: Any chance we can get https://issues.apache.org/jira/browse/MESOS-1448 too? On 3 Jul 2014, at 21:40, Vinod Kone vinodk...@gmail.com wrote: Hi, We are planning to release 0.19.1 (likely next week) which will be a bug fix release. Specifically, these are the fixes that we are planning to cherry pick. https://issues.apache.org/jira/issues/?filter=12326191jql=project%20%3D%20MESOS%20AND%20%22Target%20Version%2Fs%22%20%3D%200.19.1 If there are other critical fixes that need to be backported to 0.19.1 please reply here as soon as possible. Thanks,
Re: Mesos 0.19.0 stats.json endpoint Collectd plugin
In 0.20, stats.json is completely represented by the new endpoint. stats.json will be deprecated in one of the next few versions. On Jun 26, 2014 1:10 PM, Ray Rodriguez rayrod2...@gmail.com wrote: Hey Ben, It looks like /metrics/snapshot is a subset of the metrics exposed from /stats.json. Is stats.json eventually going to be deprecated? Why collect less metrics in this case? On Thu, Jun 26, 2014 at 3:38 PM, Ray Rodriguez rayrod2...@gmail.com wrote: Yup totally makes sense. I'll update the endpoint. Thanks. On Thu, Jun 26, 2014 at 3:33 PM, Benjamin Mahler benjamin.mah...@gmail.com wrote: This is great! If only supporting 0.19.0+ I would recommend just collecting from /metrics/snapshot because it obviates /stats.json. On Thu, Jun 26, 2014 at 12:13 PM, Ray Rodriguez rayrod2...@gmail.com wrote: Hey everyone just thought I'd post a simple collectd python plugin that I just released which turns all stats returned from the Mesos stats.json endpoint into Collectd metrics. This plugin requires Mesos 0.19.0+ and collectd 4.9+ Big shout out to phobos182(https://github.com/phobos182) who wrote the original Elasticsearch collectd python plugin upon which my plugin borrows from heavily. Mesos Collectd Python plugin: https://github.com/rayrod2030/collectd-mesos Hope this is useful to others and I welcome pull requests, comments and suggestions. Ray Rodriguez Twitter: @rayray2030 Github: rayrod2030
