Re: Apache Spark Over Mesos

>>> spark.mesos.executor.docker.image: docker-registry/mesos-spark:master-12 >>> spark.mesos.mesosExecutor.cores: 0.25 >>> spark.mesos.executor.home: /opt/spark >>> spark.mesos.uris: file:///etc/docker.tar.gz >>> >>> I am already setting it inside the doc

Re: Apache Spark Over Mesos

Hi Pradeep, Yes we still have a pending PR that will start propagating these settings down to the executors, right now it's only applied on the driver. As a work around you can download or set spark.mesos.executor.docker.image in the spark-default.conf file in the docker image you use to launch

Re: Mesos 0.27 and docker

Hi Walter, The parameters field in the container.docker is actually optional parameters that you can pass to the Docker client when you start the container, not actual command line arguments for your docker command. You should specify these in the "command" json value, either part of the value

Re: Asking for Help: Destroy docker container from marathon kills mesos slave

, 2016 at 5:41 PM, zhz shi <messi.sh...@gmail.com> wrote: > Yes we have a plan to do the upgrade but do you know the root cause of > this problem for 0.25? > > On Wed, Mar 2, 2016 at 1:49 AM, Tim Chen <t...@mesosphere.io> wrote: > >> Are you able to try out the lat

Re: Asking for Help: Destroy docker container from marathon kills mesos slave

Are you able to try out the latest Mesos release instead of 0.25? Tim On Mon, Feb 29, 2016 at 9:11 PM, shizhz wrote: > Hi all, > > Is this the correct place to ask for help? If it is could anybody help me > on the problem I posted on SOF: >

Re: Help needed (alas, urgently)

executors/ecxconfigdb.c3cae92e-baff-11e5-8afe-82f779ac6285/runs/c5c35d59-1318-4a96-b850-b0b788815f1b" >>>>> --stop_timeout="15secs" >>>>> --container="mesos-20160114-153418-1674208327-5050-3798-S0.c5c35d59-1318-4a96-b850-b0b788815f1b" >&

Re: Help needed (alas, urgently)

Hi Paul, Looks like we've already issued the docker stop as you seen in the ps output, but the containers are still running. Can you look at the Docker daemon logs and see what's going on there? And also can you also try to modify docker_stop_timeout to 0 so that we SIGKILL the containers right

Re: Mesos fetcher in dockerized slave

0157 fetcher.cpp:422] Fetcher Info: >> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/

Re: Mesos fetcher in dockerized slave

Can you share exactly how you run the slave in a docker container? Tim On Thu, Dec 17, 2015 at 1:11 PM, Marica Antonacci < marica.antona...@ba.infn.it> wrote: > No, using the socket: > > -v /var/run/docker.sock:/var/run/docker.sock > > > Il giorno 17/dic/2015, alle ore 18:07, tommy xiao

Re: Mesos fetcher in dockerized slave

feedback > Marica > > > Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <t...@mesosphere.io> ha > scritto: > > Hi Marica, > > It should work as we fetch all the files before we launch the executor and > place them in the sandbox, and we mount the san

Re: Mesos fetcher in dockerized slave

Hi Marica, It should work as we fetch all the files before we launch the executor and place them in the sandbox, and we mount the sandbox into that container as well. How did you verify that the file is not downloaded? Tim On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <

Re: Mesos fetcher in dockerized slave

Hi Shuai, You need to specify the --pid=host flag. Tim On Fri, Dec 18, 2015 at 5:19 AM, Shuai Lin wrote: > The problem happens to me if I don't specify the --docker_mesos_image > flag. However, specifying the flag only makes things worse: the task is > failed again and

Re: How does Mesos parse hadoop command??

What OS are you running this with? And I assume if you run /bin/sh and try to run hadoop it can be found in your PATH as well? Tim On Wed, Nov 4, 2015 at 12:34 AM, Du, Fan wrote: > Hi Mesos experts > > I setup a small mesos cluster with 1 master and 6 slaves, > and deploy

Re: Can't start docker container when SSL_ENABLED is on.

<user@mesos.apache.org> >> 主题: Re: Can't start docker container when SSL_ENABLED is on. >> >> I think it is easy to reproduce this error. >> >> Start master with env: >> >> SSL_SUPPORT_DOWNGRADE >> SSL_ENABLED >> SSL_KEY_FILE >> SSL_CER

Re: Can't start docker container when SSL_ENABLED is on.

Hi Xiaodong, That's the master log, but if you click on "sandbox" next to the TASK_FAILED task and find the stdout/stderr files, click on them and paste the results here. Tim On Wed, Oct 28, 2015 at 9:59 PM, Xiaodong Zhang wrote: > > The webui have a LOG link, when click it

Re: Spark Job Submitting on Mesos Cluster

aosd...@gmail.com> wrote: > >> > turn off --switch-user flag in the Mesos slave >> --no-switch_user :-) >> >> On Mon, Sep 14, 2015 at 4:03 PM, Tim Chen <t...@mesosphere.io> wrote: >> >>> Actually --proxy-user is more about which user you're impersonat

Re: Spark Job Submitting on Mesos Cluster

Actually --proxy-user is more about which user you're impersonated to run the driver, but not the user that is going to be passed to Mesos to run as. The way to use a partciular user when running a spark job is to set the SPARK_USER environment variable, and that user will be passed to Mesos.

Re: mesos-slave crashing with CHECK_SOME

Hi Scott, I wonder if you can try the latest Mesos and see if you can repro this? And if it is can you put down the example task and steps? I couldn't see disk full in your slave log so I'm not sure if it's exactly the same problem of MESOS-2684. Tim On Wed, Sep 2, 2015 at 5:15 AM, Scott

Re: Use docker start rather than docker run?

We have primitives for persistent volumes in next release (0.25.0) but DockerContainerizer integration will happen most likely the version after. Tim On Fri, Aug 28, 2015 at 11:50 AM, Tim Chen t...@mesosphere.io wrote: Hi Paul, Alternatively you can try to launch your task on the same host

Re: Use docker start rather than docker run?

. -Paul On Fri, Aug 28, 2015 at 2:26 PM, Tim Chen t...@mesosphere.io wrote: Hi Paul, We don't [re]start a container since we assume once the task terminated the container is no longer reused. In Mesos to allow tasks to reuse the same executor and handle task logic accordingly people will opt

Re: [VOTE] Release Apache Mesos 0.24.0 (rc1)

I'm working on a quick fix to fix the test, if we can just apply this we shouldn't worry about this anymore. Tim On Thu, Aug 27, 2015 at 4:27 PM, Jie Yu yujie@gmail.com wrote: Tim, maybe just remove CgroupsCpushareIsolatorProcess from CgroupsIsolatorTypes and add a TODO there for this

Re: [VOTE] Release Apache Mesos 0.24.0 (rc1)

The fix is in now, Vinod can you include the test fix? https://issues.apache.org/jira/browse/MESOS-3294 Tim On Thu, Aug 27, 2015 at 4:46 PM, Tim Chen t...@mesosphere.io wrote: I'm working on a quick fix to fix the test, if we can just apply this we shouldn't worry about this anymore. Tim

Re: Custom Scheduler: Diagnosing cause of container task failures

It received a TASK_FAILED from the executor, so you'll need to look at the sandbox logs of your task stdout and stderr files to see what went wrong. These files should be reachable by the Mesos UI. Tim On Thu, Aug 20, 2015 at 4:01 PM, Jay Taylor outtat...@gmail.com wrote: Hey everyone, I am

Re: MesosCon Seattle attendee introduction thread

Hi all, I'm a Engineer here at Mesosphere and also a Mesos PMC/Committer, for the most part working on Docker and Containerizer related things in Mesos. Looking forward to meet you all at the Hackathon and during the conference! Tim On Mon, Aug 17, 2015 at 10:30 PM, Adam Bordelon

Re: Custom docker executor

Hi Kapil, What kind of pre/post actions do you like to perform? The community has been contributing hooks that can be performed pre and post container launch, so like to see what your use cases are and perhaps the new hooks can satisfy your need, or maybe even some other way that can already do

Re: Troubles with slave recovery via Docker containerizer on 0.23.0

container is running 0.23.0 installed from packages on Ubuntu 14.04. Docker is at 1.6.2. -- b On Wed, Aug 5, 2015 at 4:28 PM, Tim Chen t...@mesosphere.io wrote: Hi Ben, Did you get the command from docker inspect or from the slave log? If it's from the slave log then we don't actually

Re: Troubles with slave recovery via Docker containerizer on 0.23.0

Hi Ben, Did you get the command from docker inspect or from the slave log? If it's from the slave log then we don't actually print out the exact way we exec the command, but just joining the exec arguments with a space in between. What's the exact error in the slave/sandbox stderr log? Tim

Re: Docker on Marathon 0.9.0 on Mesos 0.23.0

It seems like the binary (mesos-docker-executor) that was built is looking for libmesos-0.23.0 at a place where it doesn't exist. How are you running Mesos? Are you running from the source/build/src folder or after make install? Usually this happens when you don't make install before you run.

Re: Custom executor

Can you explain what your motivations are and what your new custom executor will do? Tim On Tue, Jul 28, 2015 at 5:08 AM, Aaron Carey aca...@ilm.com wrote: Hi, Is it possible to build a custom executor which is not associated with a particular scheduler framework? I want to be able to

Re: Problems connecting with Mesos Master

spark-env.sh works as it will be called by spark-submit/spark-shell, or you can just set it before you call spark-shell yourself. Tim On Tue, Jul 28, 2015 at 1:43 PM, Haripriya Ayyalasomayajula aharipriy...@gmail.com wrote: Hi, Where can I set the libprocess_ip env variable? spark_env.sh?

Re: mesos-execute + docker_image

Hi there, What kind of parameters do you like to pass to mesos-execute? You can run mesos-execute --help and it shows you all the available parameters. Tim On Tue, Jul 7, 2015 at 7:26 AM, Jürgen Jakobitsch j.jakobit...@semantic-web.at wrote: hi, i just installed mesos-0.22.0 (from the

Re: Running storm over mesos

Hi Pradeep, Without any more information it's quite impossible to know what's going on. What's in the slave logs and storm framework logs? Tim On Fri, Jul 3, 2015 at 10:06 AM, Pradeep Chhetri pradeep.chhetr...@gmail.com wrote: Hello all, I am trying to run Storm over Mesos using the

Re: service discovery in Mesos on CoreOS

As others has mentioned earlier, definitely don't use the mesos- prefix to name your docker containers since at the time we did the integration docker labels wasn't merged. Also you'll need to run mesos-slave with --pid=host, bind mount in the docker socket, and also bind mount a host directory

Re: Cluster autoscaling in Spark+Mesos ?

Hi Sharma, What metrics do you watch for demand and supply for Spark? Do you just watch node resources or you actually look at some Spark JMX stats? Tim On Thu, Jun 4, 2015 at 10:35 PM, Sharma Podila spod...@netflix.com wrote: We Autoscale our Mesos cluster in EC2 from within our framework.

Re: Cluster autoscaling in Spark+Mesos ?

Spark is aware there are more resources by getting more resource offers and using those new offers. I don't think there is a way to refresh the Spark context for streaming. Tim On Thu, Jun 4, 2015 at 1:59 PM, Dmitry Goldenberg dgoldenberg...@gmail.com wrote: Thanks, Ankur. I'd be curious to

Re: Cluster autoscaling in Spark+Mesos ?

, then resume streaming with a new instance of the Context. Would that not cause Spark to refresh its awareness of the cluster resources? - Dmitry On Thu, Jun 4, 2015 at 5:03 PM, Tim Chen t...@mesosphere.io wrote: Spark is aware there are more resources by getting more resource offers and using those

Re: Running mesos-execute inside docker.

Hi Giulio, Can you share your exact docker commands to start the mesos slave and master? Thanks! Tim On Thu, May 21, 2015 at 12:17 PM, Giulio Eulisse giulio.euli...@cern.ch wrote: Mmm, no this does not seem to work. The message is still there. Any other suggestions? -- Ciao, Giulio On

Re: Batch Scheduler with dependency support

, 2015 at 2:02 PM Tim Chen t...@mesosphere.io wrote: How are you running your batch jobs? Is the batch job script/executable an in-house app? Tim On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes andras.kere...@ishisystems.com wrote: You might want to have a look at stolos too: https

Re: Spark Bootstrapping on Mesos

Hi Stephen, I'm not quite sure what you mean by bootstrapping classes, do you have some particular examples? Usually to run any user jar you just need it to be reachable by your slaves so it can be either S3 or any accessible place, then you just provide your jar url when you run spark-submit.

Re: Batch Scheduler with dependency support

How are you running your batch jobs? Is the batch job script/executable an in-house app? Tim On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes andras.kere...@ishisystems.com wrote: You might want to have a look at stolos too: https://github.com/sailthru/stolos Andras *From:* Aaron

Re: cpu hard limit for docker containerizer?

Hi Chengwei, It's a known issue and there is a open JIRA (MESOS-2154) and also a open reviewboard that hasn't been updated for a while. I'd like this to go into to 0.23 if we can get to it, if you like to pick up the reviewboard feel free to do so. Tim On Thu, May 7, 2015 at 7:21 PM, Chengwei

Re: Kill task with configurable options?

Hi Chengwei, If you're launching tasks with Docker Containerizer, then we support a flag you can set on the slave (docker_stop_timeout) then basically does what you described. At first when you kill a docker task, we're using the docker stop command with that timeout value, which basically the

Re: Storm Mesos Error

problem? :) On Wed, Apr 29, 2015 at 1:30 PM, Tim Chen t...@mesosphere.io wrote: Hi John, Does your storm-mesos tar ball as a folder storm-mesos-0.9.3 in there? Tim On Wed, Apr 29, 2015 at 11:26 AM, John Omernik j...@omernik.com wrote: Greetings all, I got my storm nimbus running

Re: Storm Mesos Error

Hi John, Does your storm-mesos tar ball as a folder storm-mesos-0.9.3 in there? Tim On Wed, Apr 29, 2015 at 11:26 AM, John Omernik j...@omernik.com wrote: Greetings all, I got my storm nimbus running, but when I try to run a test topology, the task enters a lost state and I get the below

Re: docker based executor

Hi Tyson, The error message you saw in the logs about the executor exited actually just means the executor process has exited. Since you're launching a custom executor with MesosSupervisor, it seems like MesosSupervisor simply exited without reporting any task status. Can you look at what's the

Re: docker based executor

seems incorrect, and suggests that there is some code explicating stopping the container, instead of the container exiting itself. Thanks Tyson On Apr 18, 2015, at 3:33 AM, Tim Chen t...@mesosphere.io wrote: Hi Tyson, The error message you saw in the logs about the executor exited

Re: docker based executor

at https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L675 it mentions Either ExecutorInfo or CommandInfo should be set” Thanks Tyson On Apr 18, 2015, at 12:38 PM, Tim Chen t...@mesosphere.io wrote: That does seems odd, how did you run this via mesos? Are you using your

Re: Spark on Mesos / Executor Memory

(Adding spark user list) Hi Tom, If I understand correctly you're saying that you're running into memory problems because the scheduler is allocating too much CPUs and not enough memory to acoomodate them right? In the case of fine grain mode I don't think that's a problem since we have a fixed

Re: [RESULT][VOTE] Release Apache Mesos 0.22.0 (rc4)

Hi there, You can already pass in multiple values seperated by comma (cgroups/cpu,cgroups/mem,posix/disk) Tim On Wed, Mar 25, 2015 at 12:46 AM, Dick Davies d...@hellooperator.net wrote: Thanks Craig, that's really handy! Dumb question for the list: are there any plans to support multiple

Re: Mesos slaves connecting but not active.

How many containers are you running, and what is your system like? Also are you able to capture through perf or strace what docker rm is blocked on? Tim On Mon, Mar 23, 2015 at 10:12 AM, Giulio Eulisse giulio.euli...@cern.ch wrote: I suspect my problem is that docker rm takes forever in my

Re: mesos on coreos

Hi all, As Alex said you can run Mesos in CoreOS without Docker if you put in the dependencies in. It is a common ask though to run Mesos-slave in a Docker container in general, either on CoreOS or not. It's definitely a bit involved as you need to mount in a directory for persisting work dir

Re: spark on mesos.

Hi Dan, You won't see active frameworks happening until you start running a Spark job. This is because each Spark job actually launches a new Spark framework that is scheduling for that single job. Tim On Fri, Feb 27, 2015 at 1:39 PM, Dan Dong dongda...@gmail.com wrote: Hi, Dick, By Spark

Re: Updating FrameworkInfo settings

Mesos checkpoints the FrameworkInfo into disk, and recovers it on relaunch. I don't think we expose any API to remove the framework manually though if you really want to keep the FrameworkID. If you hit the failover timeout the framework will get removed from the master and slave. I think for

Re: preparing a host on task launch event

- result is the same - a volume/path can be provided to the docker container). Does that make a little more sense ? (a bit hard to explain). On Fri Feb 20 2015 at 1:23:46 PM Tim Chen t...@mesosphere.io wrote: Hi Michael, Can you elaborate how you use the Mesos containerizer to you prepare your

Re: Spark on Mesos Submitted from multiple users

of results etc? Lots of questions here, if these are more spark related questions, let me know, I can hop over to spark users, but since I am curious on spark on mesos, I figured I'd try here first. Thanks for your help! On Mon, Feb 16, 2015 at 10:30 AM, Tim Chen t...@mesosphere.io

Re: preparing a host on task launch event

Hi Michael, Can you elaborate how you use the Mesos containerizer to you prepare your host? In general hooks are exactly for this purpose, which is underway right now for defining the hooks in Mesos and also allowing it to be customized. Tim On Thu, Feb 19, 2015 at 6:18 PM, Michael Neale

Re: Spark on Mesos Submitted from multiple users

Hi John, With Spark on Mesos, each client (spark-submit) starts a SparkContext which initializes its own SparkUI and framework. There is a default 4040 for the Spark UI port, but if it's occupied Spark automatically tries ports incrementally for you, so your next could be 4041 if it's available.

Re: Mesos 0.22.0

Hi Dave, Sorry about the blog post, I lost track of it in the middle of other tasks. I'm going to update the website and the blog post very soon. Tim On Tue, Jan 20, 2015 at 12:37 PM, Dave Lester d...@davelester.org wrote: Thanks Niklas for kicking off this thread. +1 to you as release

Re: implementing data locality via mesos resource offers

Hi Douglas, The simplest way that Mesos can support is to add attributes via cli flags when you launch a mesos slave. And when this slave's resources is being offered, it will also include all the attributes you've tagged. This currently is static information on launch, and I believe there is

Re: implementing data locality via mesos resource offers

. On Fri, Jan 16, 2015 at 12:15 AM, Tim Chen t...@mesosphere.io wrote: Hi Douglas, The simplest way that Mesos can support is to add attributes via cli flags when you launch a mesos slave. And when this slave's resources is being offered, it will also include all the attributes you've tagged

Re: Accessing stdout/stderr of a task programmattically?

You can get the slave_id, framework_id and executor_id of a task all from state.json. ie: - { - executor_id: 20141231-115728-16777343-5050-49193-S0, - framework_id: 20141231-115728-16777343-5050-49193-, - id: 1, - labels: [ ], - name: Task 1, -

Re: concepts clarification between framework,executor, and task?

Hi Sujin, A framework can be either a long running service or just runs during the duration of the tasks. A custom executor can also run longer than the tasks themselves as well. Tasks have several states, and few them (TASK_KILLED, TASK_FAILED, TASK_FINISHED, TASK_ERROR) are states that makes

Re: Architecture question

Hi Srinivas, Can you elaborate more about what does maintaining a dynamic count of executors? You can always write a custom framework that provides the scheduling, similiar to what Marathon or Aurora is doing if it doesn't fit your need. Tim On Fri, Jan 9, 2015 at 1:18 PM, Srinivas Murthy

Re: Running services on all slaves

Hi Itamar, You can pass the amount of CPU and memory that the slave is advertising to the master for scheduling as part of the resources slave flag. So you can only schedule 12 cpus and leave 4 for your services if you want. That said, there are discussions about launching multiple tasks

Re: Running Spark on Mesos

Hi John, I'm not quite familiar how SparkSQL thrift servers are started, but in general you can't share a Mesos driver with two different frameworks in Spark. Each spark shell or spark submit creates a new framework that is independently getting offers and using these resources from Mesos. If

[RESULT][VOTE] Release Apache Mesos 0.21.1 (rc2)

Hi all, The vote for Mesos 0.21.1 (rc2) has passed with the following votes. +1 (Binding) -- Niklas Nielsen Timothy Chen Till Toenshoff +1 (Non-binding) -- Tom Arnfeld Ankur Chauhan There were no 0 or -1 votes. Please find the release

Re: [VOTE] Release Apache Mesos 0.21.1 (rc2)

On 19 December 2014 at 22:02, Tim Chen t...@mesosphere.io wrote: Hi Ankur, Since MESOS-1711 is just a minor improvement I'm inclined to include it for the next major release which shouldn't be too far away from this release. If anyone else thinks otherwise please let me know. Tim On Fri

[VOTE] Release Apache Mesos 0.21.1 (rc2)

Hi all, Please vote on releasing the following candidate as Apache Mesos 0.21.1. 0.21.1 includes the following: * This is a bug fix release. ** Bug * [MESOS-2047] Isolator cleanup failures shouldn't cause

Re: [VOTE] Release Apache Mesos 0.21.1 (rc2)

: Sorry for a late join in can we get https://issues.apache.org/jira/plugins/servlet/mobile#issue/MESOS-1711 in too or is it too late? -- ankur Sent from my iPhone On Dec 19, 2014, at 12:23, Tim Chen t...@mesosphere.io wrote: Hi all, Please vote on releasing the following candidate as Apache

[VOTE] Release Apache Mesos 0.21.1 (rc1)

Hi all, Please vote on releasing the following candidate as Apache Mesos 0.21.1. 0.21.1 includes the following: * This is a bug fix release. ** Bug * [MESOS-2047] Isolator cleanup failures shouldn't cause

Re: [VOTE] Release Apache Mesos 0.21.1 (rc1)

over the course of a failover, which is quite dangerous for production use-cases. I've attached the cherry-picks for getting these on top of 0.21.0 (not sure if apache will strip them, but you're mesosphere email should get them). On Thu, Dec 18, 2014 at 3:36 PM, Tim Chen t...@mesosphere.io

Re: DockerContainerizer error on two slaves

-working host? $ cat /proc/cgroups $ cat /proc/mounts Are you running inside a Docker or systemd container? On Tue, Dec 16, 2014 at 11:22 AM, Benjamin Mahler benjamin.mah...@gmail.com wrote: +Tim Chen (please chime in if I'm missing something) Sorry for the delay, from a quick glance

Re: Mesos slaves keep disconnecting

Is there anything in the ERROR/WARNING logs? Tim On Mon, Dec 15, 2014 at 4:22 PM, Arunabha Ghosh arunabha...@gmail.com wrote: Hi, I've setup a test mesos cluster on a few VM's running locally. I have three masters and two slaves masters : 192.168.48.14[5 - 7] slaves : 192.168.48.15[0

Re: Question about External Containerizer

Forgot to mention, unless you have a custom executor that you launch as a docker container (by putting DockerInfo in the ExecutorInfo in your TaskInfo), you can then re-use that executor for multiple tasks. Tim On Wed, Dec 3, 2014 at 11:47 AM, Tim Chen t...@mesosphere.io wrote: Hi Sharma

Re: Question about External Containerizer

Hi Sharma, Yes currently docker doesn't really support (out-of-box) launching multiple processes in the same container. They just recently added docker exec but not quite clear how it's best fit in mesos integration yet. So each task run in the Docker containerizer has to be a seperate container

Re: Timeline for 0.22.0?

Hi Scott, The patch for MESOS-1925 is already merged into master, so you should be able to just grab master in the mean time. As for 0.22.0 timeline, I don't think we set a timeline yet, usually we call a estimated time to release when we have enough to release a new version. Tim On Tue, Dec

Rocket

Hi all, Per the announcement from CoreOS about Rocket ( https://coreos.com/blog/rocket/) , it seems to be an exciting containerizer runtime that has composable isolation/components, better security and image specification/distribution. All of these design goals also fits very well into Mesos,

Re: Rocket

for the rocket toolset? Are we gonna rely on the command line interface? - Jie On Mon, Dec 1, 2014 at 11:10 AM, Tim Chen t...@mesosphere.io wrote: Hi all, Per the announcement from CoreOS about Rocket ( https://coreos.com/blog/rocket/) , it seems to be an exciting containerizer runtime that has

Re: Mesos killing Spark Driver

There are different reasons, but most commonly is when the framework ask to kill the task. Can you provide some easy repro steps/artifacts? I've been working on Spark on Mesos these days and can help try this out. Tim On Mon, Dec 1, 2014 at 2:43 PM, Gerard Maas gerard.m...@gmail.com wrote:

Re: Mesos killing Spark Driver

cause Mesos to kill the executor (not the task) to validate whether any of those conditions apply to our case and try to narrow down the problem to some reproducible subset. -kr, Gerard. On Mon, Dec 1, 2014 at 11:57 PM, Tim Chen t...@mesosphere.io wrote: There are different reasons, but most

Re: With docker containerizer enabled, How to check whether(or how) mesos successfully running tasks in docker container?

Hi Sujinzhao, Your steps s1-s3 are all correct for starting Mesos itself, but you also need a framework that can get offers from Mesos and launch Tasks. The easiest and simplest to use is the example framework that Mesos ship with (mesos-execute), or you can use richer frameworks like

Re: CFS for Docker Containers running on Mesos?

Hi Andrew, The Docker containerizer right now simply calls the docker cli cpu and memory parameters and let the Docker daemon set the cgroup shares accordingly, allow we do go behind the docker daemon to perform updates on the share amount. We didn't port the existing Mesos containerizer CFS

Re: Implementing an Executor

Hi Janet, Can you elaborate more what you like to get back from the docker container that you launched? Thanks, Tim On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the

Re: Why rely on url scheme for fetching?

-1887 - https://issues.apache.org/jira/browse/MESOS-1316 - https://issues.apache.org/jira/browse/MESOS-336 - https://issues.apache.org/jira/browse/MESOS-1248 On 31 October 2014 22:39, Tim Chen t...@mesosphere.io wrote: I believe there is already a JIRA ticket for this, if you search

Re: Why rely on url scheme for fetching?

I believe there is already a JIRA ticket for this, if you search for fetcher in Mesos JIRA I think you can find it. Tim On Fri, Oct 31, 2014 at 3:27 PM, Ankur Chauhan an...@malloc64.com wrote: Hi, I have been looking at some of the stuff around the fetcher and saw something interesting. The

Re: Exposing host services in docker container

to the hosts' localhost:27017. If that is possible, how do i do it? I started a mongos on 27017 but when i tried the command `docker run -it --rm dockerfile/mongodb bash -c 'mongo --host localhost:27017'` it was unable to connect. -- Ankur On 26 Oct 2014, at 21:12, Tim Chen t...@mesosphere.io

Re: Docker: pull on app start?

Originally we don't want to always pull with latest as that's what docker run does as well which skips pull if the image exists, and has different issues involved with this. However with MESOS-1886 we can make this optional, just figuring out where this configuration should be, at the latest

Re: Exposing host services in docker container

Hi Ankur, Not sure I understand exactly, are these common services all running on the same host where you're running the container? If it's running the same host, docker container should be able to access any port in the host, if it's cross hosts then you have to setup your own bridge and use

Re: Staging docker task KILLED after 1 minute

The case where Mesos loses track about these killed containers is going to be fixed soon, have a reviewboard up and once it merged we shouldn't have untracked containers. Tim On Fri, Oct 17, 2014 at 3:14 PM, Dick Davies d...@hellooperator.net wrote: good catch! Sorry, the docs are right I just

Re: Connecting spark from a different Machine to mesos cluster

Hi Johannes, When you started your 2nd shell, what log output from the slave do you see for that framework? Master seems to think it's already terminated. Tim On Wed, Oct 15, 2014 at 6:31 AM, Johannes Schillinger (Intern) johannes.schillin...@citrix.com wrote: Hi Tim, We are running

Re: HDFS Mesos Framework

Brenden Matthews has a HDFS framework that is still in progress: https://github.com/brndnmtthws/hdfs Welcome to contribute as well! Tim On Wed, Oct 8, 2014 at 9:51 AM, Luke Amdor luke.am...@banno.com wrote: Has anyone started work on a Hadoop HDFS Mesos framework? I know many of us just run

Re: Mesos Slave gets registered with lower memory than available

Hi Stefan, Yes it's a feature where we leave some space on each slave and not fully allocate all the memory and cpu. You can override how much resource your slave advertises by passing in the --resources flag when you start your slave. Tim On Wed, Oct 1, 2014 at 9:22 AM, Stefan Eder

Re: Mesos 0.20.1 still using -net=host when launching Docker containers

Hi Andy, The docs is sitting at the docs folder in the source tree, and there is a docker containerization doc markdown file. Simply modify it and put a patch on reviewboard, and assign to the mesos group and me. Let me know if you need more specific steps around this. Tim On Wed, Oct 1, 2014

Re: Docker executor issue

'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81' I0930 10:15:18.109361 30730 docker.cpp:1646] Executor for container 'ebb1dca6-cc9d-427f-8faa-f3f723f6ab81' has exited Thanks, Andy. -- Andy Grove VP Engineering CodeFutures Corporation On Mon, Sep 29, 2014 at 6:25 PM, Tim Chen t...@mesosphere.io wrote

Re: [VOTE] Release Apache Mesos 0.20.1 (rc3)

+1 (non-binding) Make check on Centos 5.5, docker tests all passed too. Tim On Fri, Sep 19, 2014 at 9:17 AM, Jie Yu yujie@gmail.com wrote: +1 (binding) Make check on centos5 and centos6 (gcc48) On Thu, Sep 18, 2014 at 4:05 PM, Adam Bordelon a...@mesosphere.io wrote: Hi all, Please

Re: [VOTE] Release Apache Mesos 0.20.1 (rc2)

-1 The docker test failed when I removed the image, and found a problem from the docker pull implementation. I've created a reviewboard for a fix: https://reviews.apache.org/r/25758 Will like to get this fixed before releasing it. Tim On Wed, Sep 17, 2014 at 9:10 PM, Vinod Kone

Re: Mesos 0.20.0 with Docker registry availability

Hi Maxime, It is a very valid concern and that's why I've added a patch that should go out in 0.20.1 to not do a docker pull on every run anymore. Mesos will still try to docker pull when the image isn't available locally (via docker inspect), but only once. The downside ofcourse is that you're

Re: Mesos 0.20.0 with Docker registry availability

to pull, but then does not fail solely due to the fail of a pull? In particular, we use tags to indicate which build should be deployed e.g. “foo-server:production” tag vs “foo-server:staging” tags. On Sep 4, 2014, at 11:05 PM, Tim Chen t...@mesosphere.io wrote: Hi Maxime, It is a very

Re: Launching docker containers from private repos in docker hub

The Docker Containerizer will automatically set the $HOME directory for you, so all you need is to include the .dockercfg as Michael mentioned. Thanks, Tim On Fri, Sep 5, 2014 at 10:32 AM, Michael Babineau michael.babin...@gmail.com wrote: You'll need to put a .dockercfg file somewhere the

Re: Mesos 0.20.0 with Docker registry availability

repository. So it should only pull on the first run of a given tag. ryan On 5 Sep 2014 17:41, mccraig mccraig mccraigmccr...@gmail.com wrote: hi tim, if it doesn't pull on every run, when will it pull ? :craig On 5 Sep 2014, at 07:05, Tim Chen t...@mesosphere.io wrote: Hi Maxime

  1   2   >