Re: None of the enabled containerizers (docker,mesos) could create a container for the provided TaskInfo/ExecutorInfo message.

[Adam Bordelon]

Found a likely problem in the fig.yml:
"- MESOS_ISOLATOR=cgroups/cpu,groups/mem"
That second part should be "cgroups", not "groups"

Re: implementing data locality via mesos resource offers

[Adam Bordelon]

See also the upcoming persistence primitives. Using persistent volumes (
MESOS-1554 ) and dynamic
reservations (MESOS-2018 ),
you can launch a task from one framework which creates a persistent volume
on a slave, stores its output data there, and then completes. The volume
will persist after the task exits, and its associated resources will be
offered to a framework sharing the same role. In this way, you can
guarantee data locality, and even share data between different frameworks
by placing them under the same role.

On Fri, Jan 16, 2015 at 11:43 AM, Sharma Podila  wrote:

> Hi Tim,
>
> Sure, here's some preliminary thoughts.
>
> In a Mesos cluster that has only one framework, it would suffice for the
> scheduler to have this strategy;
>
> - when assigning for a task that needs data locality, assign from an offer
> from a host that has the data
> - when assigning for a task that does need data locality, do not assign
> from an offer from a host that has/had another task which produced data
> needed by others for data locality
>
> This strategy would naturally cluster hosts into two groups: one in which
> hosts are used for data locality and another in which hosts run tasks that
> don't need data locality. Or, multiple groups if not all data is identical.
>
> Now, if there were to be multiple frameworks in the cluster, we would need
> new support in Mesos to ensure the above strategy works. Mesos allocater
> would need to do the following:
>
> - when giving out offers to framework A, prefer hosts that had other tasks
> running (or previously run) from framework A.
>
> As an example, say we have two frameworks A and B. And say there are 4
> hosts, h1, h2, h3, and h4, each with 4 cores.
> If, say, A and B are assigned 1:1, that is 8 cores each. Say currently, 2
> cores from each of the 4 hosts are offered to frameworks A and B. A variety
> of reasons could have resulted in such a split.
>
> Now, say framework A launches a task that uses 2 cores and it uses its
> offer on host h1. Now, framework A has no ability to launch another task to
> achieve data locality. To keep resource allocation still 1:1 and help data
> locality, it would be nice if Mesos did the following:
>
> - rescind 2-core offer on h1 from framework B
> - rescind 2-core offer on h2 from framework A
> - send 2-core offer on h1 to framework A
> - send 2-core offer on h2 to framework B
>
> This would need to be done only if framework A indicated, when launching
> its task on h1, that this is a task that produces data for locality
> purposes.
>
> Similarly, other scenarios and other resource types can be dealt with in
> this new strategy.
>
>
>
>
>
> On Fri, Jan 16, 2015 at 9:53 AM, Tim Chen  wrote:
>
>> Hi Sharma,
>>
>> You're correct and that's how most schedulers handle this, which is to
>> handle the locality information itself.
>>
>> We've considering and finding primitives to help in this front though, so
>> if you have any input let us know how to help manage locality that fits at
>> the level of Mesos.
>>
>> Tim
>>
>> On Fri, Jan 16, 2015 at 9:34 AM, Sharma Podila 
>> wrote:
>>
>>> Using the attributes would be the simplest way, if the slave were to
>>> support dynamic updates of the attributes. The JIRA that Tim references
>>> would be nice! Otherwise one would have to resort to something like a
>>> wrapper script of the mesos-slave process that detects new data
>>> availability and restarts mesos-slave with new attributes in cmdline.
>>> Restarts may be OK when slaves are run to checkpoint state and recover
>>> state upon restart.
>>>
>>> Another possibility in the interim would be for the framework scheduler
>>> to launch the task that does the download of the file(s) to the small
>>> subset of nodes. Then, the scheduler can maintain this state information
>>> and assign the tasks based on that. This has the additional advantage of
>>> maintaining the list of that subset of nodes in a more dynamic way, if that
>>> is useful to you.
>>>
>>> In general, I am a fan of achieving data locality via the scheduler's
>>> state info. In a more generic scenario, the data would be created
>>> dynamically by tasks previously run (instead of just an initial download)
>>> and therefore locality for such data is easier done via the scheduler.
>>>
>>>
>>>
>>> On Fri, Jan 16, 2015 at 12:15 AM, Tim Chen  wrote:
>>>
 Hi Douglas,

 The simplest way that Mesos can support is to add attributes via cli
 flags when you launch a mesos slave. And when this slave's resources is
 being offered, it will also include all the attributes you've tagged.

 This currently is static information on launch, and I believe there is
 JIRA tickets to make this dynamic (updatable at runtime).

 Tim

 On Thu, Jan 15, 2015 at 7:23 PM, Douglas Voet >>> > wrote:

> Hello,
>
> I am evaluating mesos in the context of runni

Re: Storm on Mesos, Anyone Using?

[Srinivas Murthy]

Brenden, could you please elaborate a bit on those shortcomings :-)

On Tue, Jan 20, 2015 at 11:06 AM, Brenden Matthews 
wrote:

> Hi Cory,
>
> We were using the project in production at Airbnb. It may have some
> shortcomings, but it does, in fact, work.
>
>
>> Hello all!
>> I'm interested in Storm on Mesos, but my coworkers don't wanna be guinea
>> pigs. Is anyone using mesos/storm  in
>> production? I see the repo is at least active. :)
>>
>>
>> --
>> Cory Watson
>> Principal Infrastructure Engineer // Keen IO
>
>

Re: Mesos 0.22.0

[Ankur Chauhan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

+1

On 20/01/2015 20:00, Manivannan wrote:
> +1
> 
> On Wed, Jan 21, 2015 at 7:04 AM, Vinod Kone  > wrote:
> 
> +1
> 
> @vinodkone
> 
> On Jan 20, 2015, at 12:03 PM, Chris Aniszczyk  > wrote:
> 
>> definite +1, lets keep the release rhythm going!
>> 
>> maybe some space on the wiki for release planning / release 
>> managers would be a step forward
>> 
>> On Tue, Jan 20, 2015 at 1:59 PM, Joe Stein > > wrote:
>> 
>> +1
>> 
>> so excited for the persistence primitives, awesome!
>> 
>> /*** Joe Stein Founder,
>> Principal Consultant Big Data Open Source Security LLC 
>> http://www.stealth.ly Twitter: @allthingshadoop 
>>  
>> /
>> 
>> On Tue, Jan 20, 2015 at 2:55 PM, John Pampuch > > wrote:
>> 
>> +1!
>> 
>> -John
>> 
>> 
>> On Tue, Jan 20, 2015 at 11:52 AM, Niklas Nielsen 
>> mailto:nik...@mesosphere.io>> wrote:
>> 
>>> Hi all,
>>> 
>>> We have been releasing major versions of Mesos roughly
>> every second month
>>> (current average is ~66 days) and we are now 2 months
>> after the 0.21.0
>>> release, so I would like to propose that we start
>> planning for 0.22.0
>>> Not only in terms of timing, but also because we have
>> some exciting
>>> features which are getting ready, including persistence
>> primitives, modules
>>> and SSL support (I probably forgot a ton - please chime in).
>>> 
>>> Since we are stakeholders in SSL and Modules, I would
>> like to volunteer as
>>> release manager. Like in previous releases, I'd be happy to
>>> collaborate
>> with co-release
>>> managers to make 0.22.0 a successful release.
>>> 
>>> Niklas
>>> 
>> 
>> 
>> 
>> 
>> 
>> -- Cheers,
>> 
>> Chris Aniszczyk | Open Source | Twitter, Inc. @cra | +1 512 961
>> 6719 
> 
> 
-BEGIN PGP SIGNATURE-

iQEcBAEBAgAGBQJUvyt8AAoJEOSJAMhvLp3LREwH/jsPJwSGr8ZGoi2JPv62rQsm
qwhGrmrfeKJEWijA6B8AnoUxKRVEdO65uiASpDfTVkPrrcQ+S93Pw75W6NX0oTvD
gwP8gSyDLIxAGTCkNkkxIVWueOD8Uca9s80k62dcC/LAq/2xQ9WuK+o0g2ArqVAb
035NboUM/oDYITZuauYSZCLyIRqSM6+FBpLXR0ZduEgQs5Ndq8O7FX95K+1wBmFs
yotqyQQXktlNTapMv1rCezLSYWMjX48HlKVBg3bMyamCbijcOLAk9VwH8nD+n8wA
V2JAEDt4gYorkVldOVJl+QquyWgImm7CKK43g8ZKjH5mmXKptAyFzEMh9FhloLg=
=xHRw
-END PGP SIGNATURE-

Re: Mesos 0.22.0

[Manivannan]

+1

On Wed, Jan 21, 2015 at 7:04 AM, Vinod Kone  wrote:

>  +1
>
> @vinodkone
>
> On Jan 20, 2015, at 12:03 PM, Chris Aniszczyk  wrote:
>
> definite +1, lets keep the release rhythm going!
>
> maybe some space on the wiki for release planning / release managers would
> be a step forward
>
> On Tue, Jan 20, 2015 at 1:59 PM, Joe Stein  wrote:
>
>> +1
>>
>> so excited for the persistence primitives, awesome!
>>
>> /***
>>  Joe Stein
>>  Founder, Principal Consultant
>>  Big Data Open Source Security LLC
>>  http://www.stealth.ly
>>  Twitter: @allthingshadoop 
>> /
>>
>> On Tue, Jan 20, 2015 at 2:55 PM, John Pampuch  wrote:
>>
>>> +1!
>>>
>>> -John
>>>
>>>
>>> On Tue, Jan 20, 2015 at 11:52 AM, Niklas Nielsen 
>>> wrote:
>>>
>>> > Hi all,
>>> >
>>> > We have been releasing major versions of Mesos roughly every second
>>> month
>>> > (current average is ~66 days) and we are now 2 months after the 0.21.0
>>> > release, so I would like to propose that we start planning for 0.22.0
>>> > Not only in terms of timing, but also because we have some exciting
>>> > features which are getting ready, including persistence primitives,
>>> modules
>>> > and SSL support (I probably forgot a ton - please chime in).
>>> >
>>> > Since we are stakeholders in SSL and Modules, I would like to
>>> volunteer as
>>> > release manager.
>>> > Like in previous releases, I'd be happy to collaborate with co-release
>>> > managers to make 0.22.0 a successful release.
>>> >
>>> > Niklas
>>> >
>>>
>>
>>
>
>
> --
> Cheers,
>
> Chris Aniszczyk | Open Source | Twitter, Inc.
> @cra | +1 512 961 6719
>
>

Re: Mesos 0.22.0

[Vinod Kone]

 +1

@vinodkone

> On Jan 20, 2015, at 12:03 PM, Chris Aniszczyk  wrote:
> 
> definite +1, lets keep the release rhythm going!
> 
> maybe some space on the wiki for release planning / release managers would be 
> a step forward
> 
>> On Tue, Jan 20, 2015 at 1:59 PM, Joe Stein  wrote:
>> +1
>> 
>> so excited for the persistence primitives, awesome!
>> 
>> /***
>>  Joe Stein
>>  Founder, Principal Consultant
>>  Big Data Open Source Security LLC
>>  http://www.stealth.ly
>>  Twitter: @allthingshadoop
>> /
>> 
>>> On Tue, Jan 20, 2015 at 2:55 PM, John Pampuch  wrote:
>>> +1!
>>> 
>>> -John
>>> 
>>> 
>>> On Tue, Jan 20, 2015 at 11:52 AM, Niklas Nielsen 
>>> wrote:
>>> 
>>> > Hi all,
>>> >
>>> > We have been releasing major versions of Mesos roughly every second month
>>> > (current average is ~66 days) and we are now 2 months after the 0.21.0
>>> > release, so I would like to propose that we start planning for 0.22.0
>>> > Not only in terms of timing, but also because we have some exciting
>>> > features which are getting ready, including persistence primitives, 
>>> > modules
>>> > and SSL support (I probably forgot a ton - please chime in).
>>> >
>>> > Since we are stakeholders in SSL and Modules, I would like to volunteer as
>>> > release manager.
>>> > Like in previous releases, I'd be happy to collaborate with co-release
>>> > managers to make 0.22.0 a successful release.
>>> >
>>> > Niklas
>>> >
> 
> 
> 
> -- 
> Cheers,
> 
> Chris Aniszczyk | Open Source | Twitter, Inc.
> @cra | +1 512 961 6719

Re: Mesos Community Meetings

[Vinod Kone]

On Tue, Jan 20, 2015 at 4:10 PM, Vinod Kone  wrote:

> Thanks for the interest. The next meeting will be on 5th February, 3-5 pm
> PST.
>
> The hangout link:
> https://plus.google.com/hangouts/_/twitter.com/mesos-sync
>
> On Tue, Jan 6, 2015 at 9:13 AM, Tim St Clair  wrote:
>
>> +1, also +1 re: calendar.
>>
>> --
>>
>> *From: *"David J. Palaitis" 
>> *To: *user@mesos.apache.org
>> *Sent: *Tuesday, January 6, 2015 6:02:40 AM
>> *Subject: *Re: Mesos Community Meetings
>>
>>
>> I'm also interested. Thanks for organizing.
>>
>> On Mon, Jan 5, 2015 at 11:02 PM, James DeFelice > > wrote:
>>
>>> +1 for open meetings
>>>
>>> --sent from my phone
>>> On Jan 5, 2015 10:09 PM, "Dave Lester"  wrote:
>>>
 I'm a big fan of this idea, thanks Niklas for proposing it!

 On Mon, Jan 5, 2015 at 6:27 PM, Arunabha Ghosh 
 wrote:

> +1 to hangouts, but I think hangouts has a limit of max 10 people in
> the hangout.
>
> On Mon, Jan 5, 2015 at 4:52 PM, Tom Arnfeld  wrote:
>
>> +1 also! Very interesting to hear what’s being discussed. +1 on the
>> google hangouts if these meetings are happening in person so we can 
>> listen
>> along.
>>
>>
>>
>> --
>>
>>
>> Tom Arnfeld
>>
>> Developer // DueDil
>>
>>
>>
>>
>>
>>
>> On Monday, Dec 29, 2014 at 4:12 pm, Chris Aniszczyk <
>> caniszc...@gmail.com>, wrote:
>>
>> +1 to opening up meetings! How about create a google calendar with
>> the meetings, agenda and info?
>>
>>
>> Also someone should take meeting minutes and publish them to the list
>> after each meeting for those who can't attend (on top of making 
>> information
>> more discoverable via search).
>>
>>
>>
>> Another approach is to use IRC meetings which there's a bot to record
>> meetings, but that lacks the visual aspect of GH (e.g., see IRC meeting
>> notes from Aurora:
>> http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201412.mbox/%3C20141201192131.4888419FD5%40urd.zones.apache.org%3E
>> )
>>
>>
>>
>>
>> Anyways, glad to see this finally happening.
>>
>>
>>
>>
>>
>> On Mon, Dec 29, 2014 at 7:46 AM, Niklas Nielsen 
>> wrote:Hi everyone,
>>
>>
>> Mesosphere and Twitter has been meeting up regularly to brief and
>> discuss
>>
>> current joint efforts in the Mesos project.
>>
>> While this has worked great for the engineering teams, it should be a
>>
>> community wide meeting as we discuss our agendas, timelines etc.
>> which is
>>
>> useful for a broader audience.
>>
>> Unfortunately, we cannot host people on-site, but we can open Google
>>
>> hangouts for all upcoming meetings.
>>
>>
>> Any thoughts or suggestions?
>>
>>
>> Best regards,
>>
>> Niklas
>>
>>
>>
>>
>>
>>
>> --
>> Cheers,
>>
>> Chris Aniszczyk
>> http://aniszczyk.org
>> +1 512 961 6719
>>
>
>

>>
>>
>>
>> --
>> Cheers,
>> Timothy St. Clair
>> Red Hat Inc.
>>
>
>

Re: How does the executor ID get set when using the command executor?

[Adam Bordelon]

 // Command executors share the same id as the task.
https://github.com/apache/mesos/blob/master/src/slave/slave.cpp#L2704

On Tue, Jan 20, 2015 at 2:24 PM, David Greenberg 
wrote:

> I want to know what the executor id will be set to prior to launching a
> command when using the command executor in a custom framework. From what I
> can see, if I was using a custom executor, then I could provide the
> executor ID, but I don't see a way to do so when using the command executor.
>

How does the executor ID get set when using the command executor?

[David Greenberg]

I want to know what the executor id will be set to prior to launching a
command when using the command executor in a custom framework. From what I
can see, if I was using a custom executor, then I could provide the
executor ID, but I don't see a way to do so when using the command executor.

Re: Mesos 0.22.0

[Tim Chen]

Hi Dave,

Sorry about the blog post, I lost track of it in the middle of other tasks.

I'm going to update the website and the blog post very soon.

Tim

On Tue, Jan 20, 2015 at 12:37 PM, Dave Lester  wrote:

>  Thanks Niklas for kicking off this thread. +1 to you as release manager,
> could you please create a JIRA ticket to track the progress so we could
> subscribe?
>
> A minor correction to your email, Mesos 0.21.1 was voted on in late
> December (see http://markmail.org/message/e2iam7guxukl3r6c), however the
> website wasn't updated nor was blogged about like we normally do. Tim
> (cc'd), do you still plan to make this update? Any way others can help? I'd
> like to see this updated before we cut another release.
>
> +1 to Chris' suggestion of a page to plan future release managers, this
> would bring some longer-term clarity to who is driving feature releases and
> what they include.
>
> Dave
>
> On Tue, Jan 20, 2015, at 12:03 PM, Chris Aniszczyk wrote:
>
> definite +1, lets keep the release rhythm going!
>
> maybe some space on the wiki for release planning / release managers would
> be a step forward
>
> On Tue, Jan 20, 2015 at 1:59 PM, Joe Stein  wrote:
>
> +1
>
> so excited for the persistence primitives, awesome!
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
> http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
> On Tue, Jan 20, 2015 at 2:55 PM, John Pampuch  wrote:
>
> +1!
>
> -John
>
>
> On Tue, Jan 20, 2015 at 11:52 AM, Niklas Nielsen 
> wrote:
>
> > Hi all,
> >
> > We have been releasing major versions of Mesos roughly every second month
> > (current average is ~66 days) and we are now 2 months after the 0.21.0
> > release, so I would like to propose that we start planning for 0.22.0
> > Not only in terms of timing, but also because we have some exciting
> > features which are getting ready, including persistence primitives,
> modules
> > and SSL support (I probably forgot a ton - please chime in).
> >
> > Since we are stakeholders in SSL and Modules, I would like to volunteer
> as
> > release manager.
> > Like in previous releases, I'd be happy to collaborate with co-release
> > managers to make 0.22.0 a successful release.
> >
> > Niklas
> >
>
>
>
>
>
>
> --
> Cheers,
>
> Chris Aniszczyk | Open Source | Twitter, Inc.
> @cra | +1 512 961 6719
>
>
>

Re: Mesos 0.22.0

[Dave Lester]

Thanks Niklas for kicking off this thread. +1 to you as release manager,
could you please create a JIRA ticket to track the progress so we could
subscribe?

A minor correction to your email, Mesos 0.21.1 was voted on in late
December (see http://markmail.org/message/e2iam7guxukl3r6c), however the
website wasn't updated nor was blogged about like we normally do. Tim
(cc'd), do you still plan to make this update? Any way others can help?
I'd like to see this updated before we cut another release.

+1 to Chris' suggestion of a page to plan future release managers, this
would bring some longer-term clarity to who is driving feature releases
and what they include.

Dave

On Tue, Jan 20, 2015, at 12:03 PM, Chris Aniszczyk wrote:
> definite +1, lets keep the release rhythm going!
>
> maybe some space on the wiki for release planning / release managers
> would be a step forward
>
> On Tue, Jan 20, 2015 at 1:59 PM, Joe Stein
>  wrote:
>> +1
>>
>> so excited for the persistence primitives, awesome!
>>
>> /*** Joe Stein Founder,
>> Principal Consultant Big Data Open Source Security LLC
>> http://www.stealth.ly Twitter: @allthingshadoop[1]
>> /
>>
>> On Tue, Jan 20, 2015 at 2:55 PM, John Pampuch
>>  wrote:
>>> +1!
>>>
>>> -John
>>>
>>>
>>> On Tue, Jan 20, 2015 at 11:52 AM, Niklas Nielsen
>>>  wrote:
>>>
>>> > Hi all,
>>> >
>>> > We have been releasing major versions of Mesos roughly every
>>> > second month (current average is ~66 days) and we are now 2 months
>>> > after the 0.21.0 release, so I would like to propose that we start
>>> > planning for 0.22.0 Not only in terms of timing, but also because
>>> > we have some exciting features which are getting ready, including
>>> > persistence primitives, modules and SSL support (I probably forgot
>>> > a ton - please chime in).
>>> >
>>> > Since we are stakeholders in SSL and Modules, I would like to
>>> > volunteer as release manager. Like in previous releases, I'd be
>>> > happy to collaborate with co-release managers to make 0.22.0 a
>>> > successful release.
>>> >
>>> > Niklas
>>> >
>>
>
>
>
> --
> Cheers,
>
> Chris Aniszczyk | Open Source | Twitter, Inc. @cra | +1 512 961 6719



Links:

  1. http://www.twitter.com/allthingshadoop

Re: Mesos 0.22.0

[Chris Aniszczyk]

definite +1, lets keep the release rhythm going!

maybe some space on the wiki for release planning / release managers would
be a step forward

On Tue, Jan 20, 2015 at 1:59 PM, Joe Stein  wrote:

> +1
>
> so excited for the persistence primitives, awesome!
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
> On Tue, Jan 20, 2015 at 2:55 PM, John Pampuch  wrote:
>
>> +1!
>>
>> -John
>>
>>
>> On Tue, Jan 20, 2015 at 11:52 AM, Niklas Nielsen 
>> wrote:
>>
>> > Hi all,
>> >
>> > We have been releasing major versions of Mesos roughly every second
>> month
>> > (current average is ~66 days) and we are now 2 months after the 0.21.0
>> > release, so I would like to propose that we start planning for 0.22.0
>> > Not only in terms of timing, but also because we have some exciting
>> > features which are getting ready, including persistence primitives,
>> modules
>> > and SSL support (I probably forgot a ton - please chime in).
>> >
>> > Since we are stakeholders in SSL and Modules, I would like to volunteer
>> as
>> > release manager.
>> > Like in previous releases, I'd be happy to collaborate with co-release
>> > managers to make 0.22.0 a successful release.
>> >
>> > Niklas
>> >
>>
>
>


-- 
Cheers,

Chris Aniszczyk | Open Source | Twitter, Inc.
@cra | +1 512 961 6719

Re: Mesos 0.22.0

[Joe Stein]

+1

so excited for the persistence primitives, awesome!

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/

On Tue, Jan 20, 2015 at 2:55 PM, John Pampuch  wrote:

> +1!
>
> -John
>
>
> On Tue, Jan 20, 2015 at 11:52 AM, Niklas Nielsen 
> wrote:
>
> > Hi all,
> >
> > We have been releasing major versions of Mesos roughly every second month
> > (current average is ~66 days) and we are now 2 months after the 0.21.0
> > release, so I would like to propose that we start planning for 0.22.0
> > Not only in terms of timing, but also because we have some exciting
> > features which are getting ready, including persistence primitives,
> modules
> > and SSL support (I probably forgot a ton - please chime in).
> >
> > Since we are stakeholders in SSL and Modules, I would like to volunteer
> as
> > release manager.
> > Like in previous releases, I'd be happy to collaborate with co-release
> > managers to make 0.22.0 a successful release.
> >
> > Niklas
> >
>

Re: Mesos 0.22.0

[John Pampuch]

+1!

-John


On Tue, Jan 20, 2015 at 11:52 AM, Niklas Nielsen 
wrote:

> Hi all,
>
> We have been releasing major versions of Mesos roughly every second month
> (current average is ~66 days) and we are now 2 months after the 0.21.0
> release, so I would like to propose that we start planning for 0.22.0
> Not only in terms of timing, but also because we have some exciting
> features which are getting ready, including persistence primitives, modules
> and SSL support (I probably forgot a ton - please chime in).
>
> Since we are stakeholders in SSL and Modules, I would like to volunteer as
> release manager.
> Like in previous releases, I'd be happy to collaborate with co-release
> managers to make 0.22.0 a successful release.
>
> Niklas
>

Mesos 0.22.0

[Niklas Nielsen]

Hi all,

We have been releasing major versions of Mesos roughly every second month
(current average is ~66 days) and we are now 2 months after the 0.21.0
release, so I would like to propose that we start planning for 0.22.0
Not only in terms of timing, but also because we have some exciting
features which are getting ready, including persistence primitives, modules
and SSL support (I probably forgot a ton - please chime in).

Since we are stakeholders in SSL and Modules, I would like to volunteer as
release manager.
Like in previous releases, I'd be happy to collaborate with co-release
managers to make 0.22.0 a successful release.

Niklas

Re: Storm on Mesos, Anyone Using?

[Brenden Matthews]

Hi Cory,

We were using the project in production at Airbnb. It may have some
shortcomings, but it does, in fact, work.


> Hello all!
> I'm interested in Storm on Mesos, but my coworkers don't wanna be guinea
> pigs. Is anyone using mesos/storm  in
> production? I see the repo is at least active. :)
>
>
> --
> Cory Watson
> Principal Infrastructure Engineer // Keen IO

Spark 1.20 resource issue with Mesos .21.1

[Brian Belgodere]

Hi All,
I'm running into a weird issue with my test mesos cluster, I have a 3
master / 3 slave HA configuration. Marathon and Chronos are working as they
should and I can deploy dockerized applications to the slave nodes without
issue using Marathon. I downloaded Spark 1.2 and built from source.
Standalone mode works correctly but when I attempt to submit jobs to the
Mesos Cluster from Spark, it connects and shows up as a framework but I get
"Initial job has not accepted any resources; check your cluster UI to
ensure that workers are registered and have sufficient memory". I have
appended the relevant info believe below and I appreciate any help with
this. I've tried this in both coarse and fine grain and I get the same
result.

-Brian

I'm running on ubuntu trusty 64

my spark-env.sh contains

export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
export SPARK_EXECUTOR_URI=http://192.0.3.11:8081/spark-1.2.0.tgz
export MASTER=mesos://zk://192.0.3.11:2181,192.0.3.12:2181,
192.0.3.13:2181/mesos
export SPARK_WORKER_MEMORY=512M
export SPARK_WORKER_CORES=1
export SPARK_LOCAL_IP=192.0.3.11

My Mesos Cluster sees

*Cluster*:
Mesos_Cluster
*Server*:
192.0.3.12:5050
*Version*:
0.21.1
*Built*:
a week ago by root
*Started*:
2 hours ago
*Elected*:
2 hours ago

*Resources*


*CPUs*

*Mem*

*Total*

3

2.9 GB

*Used*

0

0 B

*Offered*

0

0 B

*Idle*

3

2.9 GB


In the Spark Log I see

vagrant@master1:~/spark-1.2.0$ ./bin/run-example SparkPi 3
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
15/01/19 02:41:40 INFO SecurityManager: Changing view acls to: vagrant
15/01/19 02:41:40 INFO SecurityManager: Changing modify acls to: vagrant
15/01/19 02:41:40 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(vagrant);
users with modify permissions: Set(vagrant)
15/01/19 02:41:41 INFO Slf4jLogger: Slf4jLogger started
15/01/19 02:41:41 INFO Remoting: Starting remoting
15/01/19 02:41:42 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@master1:56626]
15/01/19 02:41:42 INFO Utils: Successfully started service 'sparkDriver' on
port 56626.
15/01/19 02:41:42 INFO SparkEnv: Registering MapOutputTracker
15/01/19 02:41:42 INFO SparkEnv: Registering BlockManagerMaster
15/01/19 02:41:42 INFO DiskBlockManager: Created local directory at
/tmp/spark-local-20150119024142-16af
15/01/19 02:41:42 INFO MemoryStore: MemoryStore started with capacity 267.3
MB
15/01/19 02:41:42 INFO HttpFileServer: HTTP File server directory is
/tmp/spark-80342d7e-780f-4550-933d-adce88265322
15/01/19 02:41:42 INFO HttpServer: Starting HTTP Server
15/01/19 02:41:42 INFO Utils: Successfully started service 'HTTP file
server' on port 36273.
15/01/19 02:41:43 INFO Utils: Successfully started service 'SparkUI' on
port 4040.
15/01/19 02:41:43 INFO SparkUI: Started SparkUI at http://master1:4040
15/01/19 02:41:43 INFO SparkContext: Added JAR
file:/home/vagrant/spark-1.2.0/examples/target/scala-2.10/spark-examples-1.2.0-hadoop1.0.4.jar
at http://192.0.3.11:36273/jars/spark-examples-1.2.0-hadoop1.0.4.jar with
timestamp 1421635303639
2015-01-19 02:41:44,069:19208(0x7f7da54b3700):ZOO_INFO@log_env@712: Client
environment:zookeeper.version=zookeeper C client 3.4.5
2015-01-19 02:41:44,070:19208(0x7f7da54b3700):ZOO_INFO@log_env@716: Client
environment:host.name=master1
2015-01-19 02:41:44,070:19208(0x7f7da54b3700):ZOO_INFO@log_env@723: Client
environment:os.name=Linux
2015-01-19 02:41:44,071:19208(0x7f7da54b3700):ZOO_INFO@log_env@724: Client
environment:os.arch=3.13.0-43-generic
2015-01-19 02:41:44,071:19208(0x7f7da54b3700):ZOO_INFO@log_env@725: Client
environment:os.version=#72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014
2015-01-19 02:41:44,072:19208(0x7f7da54b3700):ZOO_INFO@log_env@733: Client
environment:user.name=vagrant
2015-01-19 02:41:44,072:19208(0x7f7da54b3700):ZOO_INFO@log_env@741: Client
environment:user.home=/home/vagrant
2015-01-19 02:41:44,073:19208(0x7f7da54b3700):ZOO_INFO@log_env@753: Client
environment:user.dir=/home/vagrant/spark-1.2.0
2015-01-19 02:41:44,073:19208(0x7f7da54b3700):ZOO_INFO@zookeeper_init@786:
Initiating client connection, host=192.0.3.11:2181,192.0.3.12:2181,
192.0.3.13:2181sessionTimeout=1 watcher=0x7f7daa4516a0 sessionId=0
sessionPasswd= context=0xcf0a60 flags=0
2015-01-19 02:41:44,077:19208(0x7f7da3cb0700):ZOO_INFO@check_events@1703:
initiated connection to server [192.0.3.13:2181]
2015-01-19 02:41:44,080:19208(0x7f7da3cb0700):ZOO_INFO@check_events@1750:
session establishment complete on server [192.0.3.13:2181],
sessionId=0x34aff9e627f000e, negotiated timeout=1
I0119 02:41:44.082293 19313 sched.cpp:137] Version: 0.21.1
I0119 02:41:44.088546 19315 group.cpp:313] Group process (group(1)@
192.0.3.11:50317) connected to ZooKeeper
I0119 02:41:44.088948 19315 group.cpp:790] Syncing group operations: queue
size (joins, cancels, datas) = (0, 0, 0)
I0119 02:41:44.089274 19315 group.cpp:385] Trying to create path '/mesos'
in

Storm on Mesos, Anyone Using?

[Cory Watson]

Hello all!

I'm interested in Storm on Mesos, but my coworkers don't wanna be guinea
pigs. Is anyone using mesos/storm  in
production? I see the repo is at least active. :)

-- 
Cory Watson
Principal Infrastructure Engineer // Keen IO