Re: what's the difference between mesos and yarn?

Checkout Apache Myriad: You can run Yarn on Mesos :) On Mon, Aug 15, 2016 at 10:35 PM, tommy xiao wrote: > yarn is based on BigData community to provide resource manager, and mesos > is general datacenter focus resource manager. it's feature have some > overlap and the

Dynamic Reservations and Roles

Hey all, I am trying to come up with a process that I can say "I am running as "prod" principal, I connect to the reserve endpoint, and I make a request for X CPU, Y Mem,, for the "dev role and usable by the "dev" principal. I feel like that I should be able to reserve that out, i.e. as a prod

Re: Access to Design Doc

I have access now, I will review. Thanks On Tuesday, January 12, 2016, Greg Mann <g...@mesosphere.io> wrote: > Hi John, > I just shared the doc with you; let me know if you still have trouble > accessing it. > > Cheers, > Greg > > On Tue, Jan 12, 2016 at 11:42 AM,

Access to Design Doc

Is there a place to request google doc permissions on the design doc here: https://issues.apache.org/jira/browse/MESOS-2840

Re: mesos-elasticsearch vs Elasticsearch with Marathon

I'll just toss another way.. there is an elastic search on yarn framework that actually works really nice with Apache Myriad (for running Yarn on Mesos) I know it sounds a bit convoluted but I have set it up so I can create ES clusters on demand, just give me a cluster name, and node size, and it

Re: Dynamic Reservations and --roles

I filed https://issues.apache.org/jira/browse/MESOS-4143 to address the reserve unreserve endpoints reserving roles for non-explicit rules. I also opened https://issues.apache.org/jira/browse/MESOS-4144 to add the ability to add roles dynamically based on an API request to properly authorized

Dynamic Reservations - API to See them

Is there an API endpoint that allows an operator to see the current dynamic reservations? I keep track of what's there etc. John

Dynamic Reservations and --roles

I am following http://mesos.apache.org/documentation/latest/reservation/ to learn how to do dynamic reservations. I don't want to statically assign roles, therefore, I started my slaves only with the (*) resources. I used the HTTP endpoints to reserve some resources. And it returned successful.

Mesos ACL User

In crafting my ACLs, I found that I would like to have a situation where groups were used instead of just user... i.e. if I have a certain frame, perhaps a dev instance of Marathon, I want folks in the dev group to all be able to to run frameworks as themselves. Right now, have a principal that

Weird Error on One Node

I am trying to start a task on a node, and it keeps failing with no logs in the sandbox (it's blank) In the slave logs, I get the error below. I looked into MESOS-3352 and since I am running RHEL 7 with Version 208 (that "should" be the patched version based on what I've read). I am thinking

Re: Weird Error on One Node

I tried restarting the slave again and this time it worked... *shrug* That said, anything I can do next time this happens? Do I need to do something on RHEL7 and Cgroups to make the info in MESOS-3352 work? Thanks! John On Fri, Dec 4, 2015 at 1:54 PM, John Omernik <j...@omernik.com>

Re: Web UI Memory Usage in Firefox

os JIRA. Thank you in advance. > > Best > Orlando > > On Wednesday, December 2, 2015 at 8:26:04 PM UTC+1, John Omernik wrote: >> >> Marathon Issue filed. >> >> https://github.com/mesosphere/marathon/issues/2755 >> >> At this point, should I look at a

Re: Native Lib vs. Rest API

che.org/jira/browse/MESOS-3302> > > The user doc for the scheduler API is > https://github.com/apache/mesos/blob/master/docs/scheduler-http-api.md > > On Thu, Dec 3, 2015 at 11:34 AM, John Omernik <j...@omernik.com> wrote: > >> Somewhere in the back of my brain I thought

Native Lib vs. Rest API

Somewhere in the back of my brain I thought I read something about a migration away from using the mesos native lib and going to a more generic API approach to support better portability and less reliance on the lib. I read about this before I understood things well (or as well I do now I should

Roles and Oversubscription

How do roles and oversubscription work together? When you specify resources that can be oversubscribed, do you say what role they work in? Is "revocable" a role in and of itself? I am trying to work through these various items in my path to learning more about oversubscription. Thanks! John

Web UI Memory Usage in Firefox

I am cross posting this in Marathon and Mesos lists because both UIs are having this issue, and I figured I'd save time in posting two separate messages. Basically, in using Firefox, I noticed that over time, my firefox would get to become unusable when I had Marathon and Mesos WebUIs up and

Re: Web UI Memory Usage in Firefox

ating to about:memory and clicking the "GC" button? The > web UI's definitely should not need that much memory. > > ~Joseph > > On Wed, Dec 2, 2015 at 9:07 AM, John Omernik <j...@omernik.com> wrote: > >> I am cross posting this in Marathon and Mesos lists because b

Re: Web UI Memory Usage in Firefox

> > On Wednesday, December 2, 2015 at 6:07:40 PM UTC+1, John Omernik wrote: >> >> I am cross posting this in Marathon and Mesos lists because both UIs are >> having this issue, and I figured I'd save time in posting two separate >> messages. >> >> Basically,

Re: Docker Multi-Host Networking and Mesos Isolation Strategies

dule and let user choose > which network isolator should used. But this is just my opinion. Could you > create a story for this in https://issues.apache.org/jira/browse/MESOS so > we could track this better. > > On Wed, Nov 4, 2015 at 9:29 PM, John Omernik <j...@omernik.com>

Re: Marathon 0.11.1 - Mesos 0.25 - Mesos-DNS 0.4.0

lt;james.defel...@gmail.com> wrote: > What settings worked for you? We did aim for least surprise. Sounds like > we missed a bit. We're happy to accept suggestions for improvement via gh > issues filed against the mesos-dns repo. > On Oct 29, 2015 7:39 AM, "John Omernik" &

Re: Marathon 0.11.1 - Mesos 0.25 - Mesos-DNS 0.4.0

should > have been the fallback. > > Did you, by chance, have `docker` listed in IPSources at any point? > > > On Tue, Nov 3, 2015 at 12:04 PM, John Omernik <j...@omernik.com> wrote: > >> I used >> >> "IPSources": ["host", "netinfo"

Re: Marathon 0.11.1 - Mesos 0.25 - Mesos-DNS 0.4.0

the change. On Wed, Oct 28, 2015 at 3:02 PM, Grzegorz Graczyk <gregor...@gmail.com> wrote: > It's not a bug, it's a feature - > http://mesosphere.github.io/mesos-dns/docs/configuration-parameters.html look > at IPSources config > > śr., 28.10.2015 o 15:59 użytkownik John Omer

Marathon 0.11.1 - Mesos 0.25 - Mesos-DNS 0.4.0

. Is there a setting that I need to use the external IP of the container? Is this issue known? Is there a workaround? This is pretty major for Docker running on Marathon and using Mesos-dns for service discovery. John Omernik

Re: Marathon 0.11.1 - Mesos 0.25 - Mesos-DNS 0.4.0

If I rolled back mesos-dns to v0.2.0 (on the releases page) then it pulls the right IP address.. (Mesos-dns version is the easiest of the three to change) John On Wed, Oct 28, 2015 at 9:52 AM, John Omernik <j...@omernik.com> wrote: > So, the issues that are listed appear to be

Re: Marathon 0.11.1 - Mesos 0.25 - Mesos-DNS 0.4.0

os 0.25.0 because of issues mentioned > in the mesos mailing list regarding Marathon 0.11.x and Mesos 0.25.0 > > On Wed, Oct 28, 2015 at 10:38 AM, John Omernik <j...@omernik.com> wrote: > >> Hey all - >> >> I am cross posting this because it's a number of moving parts th

Re: Odd Scenerio with Mesos.

On it, it's in a weird PoC lab thing and I have to do some gyrations to get logs off, it will be soon. On Wed, Oct 21, 2015 at 2:46 PM, Vinod Kone <vinodk...@gmail.com> wrote: > Logs please. > > On Wed, Oct 21, 2015 at 12:44 PM, John Omernik <j...@omernik.com> wrote: &

Odd Scenerio with Mesos.

I am running 0.24. I am running some tasks in marathon, and when they hit an OOM condition a task is killed that is expected. Than I get a bunch of errors related to "Failed to read "meory.limit_in_bytes', 'memory.max_usage_in_bytes' and memory.stat. In addition the task tries to restart but

Mesos-Dns Masters List

Hey all, I was using mesos-dns, and I filled in my zk field based on the HA mesos cluster I have. Mesos dns is up, but in the stderr, I keep seeing "generator.go:342 warning: leader "master@10.0.0.1:5050" is not in master list. I don't have a master list in my config.json, instead I am using

Mesos/Marathon/HAProxy Logging

I have been playing with an application that is a very simple app: A webservice running in Python. I've created a docker container, it runs in the container, I setup marathon to run it, I use mesos-dns and ha proxy and I can access the service just fine anywhere in the cluster. First let me say

Re: Mesos/Marathon/HAProxy Logging

reverse proxies and this is how we log the actual source clients to solve similar auditing and compliance needs. On Tuesday, August 25, 2015, John Omernik j...@omernik.com wrote: I have been playing with an application that is a very simple app: A webservice running in Python. I've created

Re: Mesos Modifying User Group

I ran into this same issue. For me it manifested as weird permission denied in MapR's NFS implementation, running in bash, etc was fine. But running in on Mesos, it didn't work (permission denied)(Also thank you to MapR for helping me troubleshoot). Good news, there is a patch.

Re: Static Resource Reservation on mesos slave

I am reviving this thread as I am looking to do something similar but have a question --resources=cpus(role1):1;mem(role1):2048;disk(role1):32768;cpus(*):11 In what Vinod has put works, but in reality I want (assuming a 12 vcore and 32768 MB of ram node)

--resources documentation

Hey, all, I am looking to set my slave resources and was looking at the documentation and was unclear exactly the format used by memory (and/or disk) I am going to assume based on the numbers below, that 15360 is like in MB? (15 GB of ram seems like a good example, as opposed to saying it's in KB

Docker on Marathon 0.9.0 on Mesos 0.23.0

I am finding that Docker Containers won't start for me in the versions above, the only information I am getting from the sandbox is below, I am not sure what the issue is in that the file is in the same location where the previous version files were... Any help is appreciated. John

Re: --resources documentation

, Greg On Tue, Aug 4, 2015 at 6:54 AM, John Omernik j...@omernik.com wrote: Hey, all, I am looking to set my slave resources and was looking at the documentation and was unclear exactly the format used by memory (and/or disk) I am going to assume based on the numbers below, that 15360 is like

Re: Build 0.23 gcc Version

So, I don't mean to sound like a newbie here, but in running my current setup which has 4.6.3, (and I tried to run 4.8) how can I get Mesos 0.23 to compile. Is this something I need to change in certain files? In certain steps? Is this something that should be a bug in Mesos to handle the

Re: Build 0.23 gcc Version

. Ben. On Fri, Jul 24, 2015 at 2:02 PM John Omernik j...@omernik.com wrote: I am trying to build 0.23, I got the error below. I already installed gcc-4.8 and set my alternatives to work with 4.8 as you can see gcc --version returns the right version, where is the configure script pulling

Build 0.23 gcc Version

I am trying to build 0.23, I got the error below. I already installed gcc-4.8 and set my alternatives to work with 4.8 as you can see gcc --version returns the right version, where is the configure script pulling that data? Are there flags I could use to help it through the process? :) John

Mailing lists for specific frameworks

Hey all, many of the frameworks in development such as the mesos-kafka and mesos-elasticsearch look very promising! In order to keep the Mesos users group clean, I was wondering if there are mailing lists, google groups etc set for some of these. I think it could be beneficial from a standpoint

Re: Writing outside the sandbox

membership of darkness falls into that? Regards, Bjoern On May 12, 2015, at 5:53 AM, John Omernik j...@omernik.com wrote: So I tried su darkness and su - darkness and both allowed a file write with no issues. On the group thing, while it is weird would that actually hurt ti to contain

Re: Writing outside the sandbox

. *Marco Massenzio* *Distributed Systems Engineer* On Mon, May 11, 2015 at 6:54 AM, John Omernik j...@omernik.com wrote: Paul: I checked in multiple places and I don't see rootsquash being used. I am using the MapR NFS server, and I do not believe that is a common option in the default setup ( I

Re: Writing outside the sandbox

differences when run via Marathon vs. directly. Maybe you're running in a different shell? On Sun, May 10, 2015 at 2:21 PM, John Omernik j...@omernik.com wrote: I believe the slave IS running as root. FWIW when I ran the script from above as root, it did work as intended (created the files

Re: Writing outside the sandbox

I am not specifying isolators. The Default? :) Is that a per slave setting? On Sat, May 9, 2015 at 3:33 PM, James DeFelice james.defel...@gmail.com wrote: What isolators are you using? On Sat, May 9, 2015 at 3:48 PM, John Omernik j...@omernik.com wrote: Marco... great idea... thank you. I

Writing outside the sandbox

I am doing something where people may recommend against my course of action. However, I am curious if there is a way basically I have a process being kicked off in marathon that is trying to write to a nfs location. The permissions of the user running the task and the nfs location are good. So

Re: Storm Mesos Error

, Apr 29, 2015 at 11:26 AM, John Omernik j...@omernik.com wrote: Greetings all, I got my storm nimbus running, but when I try to run a test topology, the task enters a lost state and I get the below in my stderr on the sandbox. Note, the URL for the storm.yaml works fine, not sure why it's

Storm Mesos Error

Greetings all, I got my storm nimbus running, but when I try to run a test topology, the task enters a lost state and I get the below in my stderr on the sandbox. Note, the URL for the storm.yaml works fine, not sure why it's causing an issue on the cp. cp: cannot create regular file

Re: Storm Mesos Error

, John Omernik j...@omernik.com wrote: I used the bin/build-release.sh package and it put in all in a folder named apache-storm-0.9.3... that's probably my problem? :) On Wed, Apr 29, 2015 at 1:30 PM, Tim Chen t...@mesosphere.io wrote: Hi John, Does your storm-mesos tar ball as a folder storm

Re: Using mesos-dns in an enterprise

, John Omernik j...@omernik.com wrote: That was my first response as well... I work at a bank, and the thought of changing dns servers on the clients everywhere made me roll my eyes :) John On Thu, Apr 2, 2015 at 9:39 AM, Tom Arnfeld t...@duedil.com wrote: This is great, thanks for sharing

Using mesos-dns in an enterprise

Based on my earlier emails about the state of service discovery. I did some research and a little writeup on how to use mesos-dns as a forward lookup zone in a enterprise bind installation. I feel this is more secure, and more comfortable for an enterprise DNS team as opposed to changing the

Re: Using mesos-dns in an enterprise

integrated consul dns at client sites. Bind config still needs updating if/when mesos dns relocates. --sent from my phone On Apr 2, 2015 10:30 AM, John Omernik j...@omernik.com wrote: Based on my earlier emails about the state of service discovery. I did some research and a little writeup on how

Current State of Service Discovery

I have been researching service discovery on Mesos quite a bit lately, and due to my background, may be making assumptions that don't apply to a Mesos Datacenter. I've read through docs, and I have come up with two main approaches to service discovery, and both appear to have strengths and

HAProxy for Hive Thrift Server on Mesos

I have a nice setup with a Hive thrift server running in a docker container on Mesos. It works pretty well, but something, I believe in how HAProxy works with the connection, is causing the thrift server connection to die after a time. Basically, I can run a few queries, but after 2 or 3, or

Running Spark on Mesos

I have Spark 1.2 running nicely with both the SparkSQL thrift server and running it in iPython. My question is this. I am running on Mesos in fine grained mode, what is the appropriate way to manage the two instances? Should I run a Course grained mode for the Spark SQL Thrift Server so that RDDs

Passing -D Java Options to hadoop-mesos

I'd like to pass some -D options to my java instance running hadoop on hadoop-mesos. Where can I set that up to be properly passed through Mesos? Thanks! John

Re: hadoop-mesos error

and the configuration that gets printed out by the executor? Also, what version of Hadoop are you running, and what revision of the framework? Cheers, Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Tue, Nov 18, 2014 at 8:27 PM, John Omernik j

Re: Untaring Framework tgzs: Can we customize?

. We'll add the option to CommandInfo.URI so that frameworks can override the default if desired. Mind filing a ticket? On Thu, Sep 11, 2014 at 5:00 AM, John Omernik j...@omernik.com wrote: Vinod - I believe this is EXACTLY the issue. I also understand why in most cases this is ok. If a user

Untaring Framework tgzs: Can we customize?

I am wondering about the process of fetching the tgz files and running them on slaves. Basically, I am trying to run hadoop-mesos, but still use the LinuxTaskController (http://hadoop.apache.org/docs/r1.0.4/cluster_setup.html for details). When I am using hadoop, I have to swich to the

Re: Sandbox Log Links

Ya, just confirmed, when I set --work_dir=anything (anything being even the default /tmp/mesos/slave/ ) there are no sandbox logs, yet, when I leave it off on the slave, then it shows the sandbox. Anything thoughts? Anyone able to reproduce? On Thu, Sep 4, 2014 at 7:23 PM, John Omernik j

Re: Sandbox Log Links

that was part of your task failure. Tim On Thu, Sep 4, 2014 at 3:24 PM, John Omernik j...@omernik.com wrote: Hey all, I upgraded to 0.20 and when I click on sandbox, the link is good, but there are not futher links for logs (i.e. standard err, out etc) like there was in 0.19. I have changed my

Re: Struggling with task controller Permissions on Hadoop Mesos

Just to keep all updated: The issue is MapR uses the LinuxTaskController by default. I went back to the DefaultTaskController, and this fixed my issue! Thanks! http://hadoop.apache.org/docs/r1.0.4/cluster_setup.html On Tue, Aug 19, 2014 at 1:47 PM, John Omernik j...@omernik.com wrote: Well

Data Center Awareness - Mesos

I was wondering... does Mesos have any concept of datacenter awareness? I.e. if you have two primary data centers, nodes can be flagged as such, and then certain frameworks can be localized to a datacenter, or, if the frameworks allow, be distributed across high latency links? Or is this all just

Re: Struggling with task controller Permissions on Hadoop Mesos

Yes On Monday, August 18, 2014, Vinod Kone vinodk...@gmail.com wrote: On Sat, Aug 16, 2014 at 4:26 AM, John Omernik j...@omernik.com javascript:_e(%7B%7D,'cvml','j...@omernik.com'); wrote: I've confirmed on the package I am using that when I untar it using tar zxf as root, that the task

Re: Struggling with task controller Permissions on Hadoop Mesos

Can you do me a favor? on one of your running tasks, or recently completed tasks, in the Mesos task, click on it, go to the logs (it shows the stderr and stdout) and then drill into the extracted hadoop package to /hadoop-version/bin/Linux-amd64-64/bin and let me know what the owner/permissions of

Re: Alternate HDFS Filesystems + Hadoop on Mesos

-0.20.2-mapr-4.0.0.tgz Failed to synchronize with slave (it's probably exited) On Fri, Aug 15, 2014 at 5:38 PM, John Omernik j...@omernik.com wrote: I am away from my cluster right now, I trued doing a hadoop fs -ls maprfs:// and that worked. When I tries hadoop fs -ls hdfs:/// it failed

Re: Alternate HDFS Filesystems + Hadoop on Mesos

, 2014 at 5:38 PM, John Omernik j...@omernik.com wrote: I am away from my cluster right now, I trued doing a hadoop fs -ls maprfs:// and that worked. When I tries hadoop fs -ls hdfs:/// it failed with wrong fs type. With that error I didn't try it in the mapred-site. I will try it. Still

Re: Does Mesos support Hadoop MR V2

So excuse my naivety in this space, but my ignorance has never really stopped me from asking questions: I see YARN (Yet another resource negotiator) as very similar to Mesos. I.e. something to manage resources on a cluster of machines. So when I hear talk of running YARN on Mesos it's seems very