Re: Could mesos be a replacement for yarn?

[Klaus Ma]

I don-t think so :(. There's Apache incubator project, named Myriad, can
run YARN on Mesos, but it's far away from production.

On Wed, Dec 21, 2016 at 12:01 AM Dima Fadeyev  wrote:

> Hello, everyone,
>
> Is it possible to run software from hadoop ecosystem on mesos? Do these
> work: Hive, Oozie, Sqoop, MRv2, Pig?
>
> Thanks in advance and best regards
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: Welcome Haosdent Huang as Mesos Committer and PMC member!

[Aaron Carey]

Congratulations! Very well deserved! Always so helpful :)

Aaron Carey
Production Engineer - Cloud Pipeline
Industrial Light & Magic
London
020 3751 9150


On 20 December 2016 at 01:24, Jie Yu  wrote:

> Congrats! Well deserved!!
>
> Always wondering why you have so much time!
>
> - Jie
>
> On Mon, Dec 19, 2016 at 5:19 PM, Jay Guo  wrote:
>
>> Congratulations Haosdent!!!
>>
>> /J
>>
>> On Mon, Dec 19, 2016 at 4:40 PM, Chengwei Yang
>>  wrote:
>> > Congratulations! Well deserved.
>> >
>> > Haosdent helps me a lot!
>> >
>> > On Fri, Dec 16, 2016 at 01:59:19PM -0500, Vinod Kone wrote:
>> >> Hi folks,
>> >>
>> >> Please join me in formally welcoming Haosdent Huang as Mesos Committer
>> and
>> >> PMC member.
>> >>
>> >> Haosdent has been an active contributor to the project for more than a
>> year
>> >> now. He has contributed a number of patches and features to the Mesos
>> code
>> >> base, most notably the unified cgroups isolator and health check
>> >> improvements. The most impressive thing about him is that he always
>> >> volunteers to help out people in the community, be it on slack/IRC or
>> >> mailing lists. The fact that he does all this even though working on
>> Mesos
>> >> is not part of his day job is even more impressive.
>> >>
>> >> Here is his more formal checklist
>> >> > H8ZGLXG6CF9VP2IY_UU5_0/edit?ts=57e0029d>
>> >> for your perusal.
>> >>
>> >> Thanks,
>> >> Vinod
>> >>
>> >> P.S: Sorry for the delay in sending the welcome email.
>> >
>> > --
>> > Thanks,
>> > Chengwei
>>
>
>

Re: how to debug when a task is killed

[Luke Adolph]

My app does not provide health check mechanism.
And I have got the reason: *Not enough memory*.
I should provide more memory for my app on Marathon.
Thanks all!
​

2016-12-20 15:17 GMT+08:00 haosdent :

> Do you configure health check? If you configure health check and it could
> not pass, the task would be killed.
>
> On Tue, Dec 20, 2016 at 2:23 PM, Luke Adolph  wrote:
>
>> Hi all:
>>
>> I have set up a mesos cluster with on mesos master and five mesos agents.
>> I use Marathon to depoy an app across mesos agents, which reads process
>> info from /proc.
>> About every 40 minuntes, my apps will be killed and Marathon restart them.
>> The stderr info in sandbox is:
>> ​
>>
>> I1220 05:05:12.014192 28736 exec.cpp:143] Version: 0.28.1
>> I1220 05:05:12.017397 28740 exec.cpp:217] Executor registered on slave 
>> 83e33a06-5794-4baa-a654-dd2ecfcd426d-S5
>> 2016/12/20 05:05:12 status read fail.
>> 2016/12/20 05:05:12 process id is: 8208
>> 2016/12/20 05:05:12 open /proc/8208/status: no such file or directory
>> 2016/12/20 05:06:16 status read fail.
>> 2016/12/20 05:06:16 process id is: 8742
>> 2016/12/20 05:06:16 open /proc/8742/status: no such file or directory
>> 2016/12/20 05:07:16 status read fail.
>> 2016/12/20 05:07:16 process id is: 9005
>> 2016/12/20 05:07:16 open /proc/9005/status: no such file or directory
>> 2016/12/20 05:25:50 status read fail.
>> 2016/12/20 05:25:50 open /proc/17284/stat: no such file or directory
>> Killed
>>
>> ​
>>
>> In addition to above stderr info, I have no meaningful info to provide or
>> debug.
>> May you share your experience on solving similar situation.
>>
>> Thanks very much！
>>
>> --
>> Thanks & Best Regards
>> 卢文泉 | Adolph Lu
>> TEL：+86 15651006559 <+86%20156%205100%206559>
>> Linker Networks(http://www.linkernetworks.com/)
>>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>



-- 
Thanks & Best Regards
卢文泉 | Adolph Lu
TEL：+86 15651006559
Linker Networks(http://www.linkernetworks.com/)

Re: Mesos on AWS

[Kiril Menshikov]

​Hey,

Sorry for delayed response. I reinstalled my AWS infrastructure. Now I
install everything on RedHat linux. Before I use Amazon Linux.

I tested with single master (m4.large). Everything works perfect. I am not
sure if it was Amazon Linux or my old configurations.

Thanks,
​-Kirils

On 18 December 2016 at 14:03, Guillermo Rodriguez 
wrote:

> Hi,
> I run my mesos cluster in AWS, betewwn 40 to 100 m4.2xlarge instances at
> any time. Between 200 and 1500 jobs anytime. Slaves run as spot instances.
>
> So, the only moment I get a TASK_LOST is when I lose a spot instance due
> to being outbid.
>
> I guess you may also lose instances due to an AWS autoscaler scale-in
> procedure, for example, if it decides the cluster is inderutilised then it
> can kill any instane in your cluster, not necessarilly the least used one.
> That's the reason we decided to develop our customised autoscaler that
> detects and kills specific instances based on our own rules.
>
> So, are you using spot fleets or spot innstances? Have you setup your
> scale-in procedures correctly?
>
> Also, if you are running fine grained tiny jobs (400 jobs in a 10xlarge
> means 0.1 CPUs and 400MB RAM each), I recommend you avoid an m4.10xlarge
> instance and run xlarge instances instead. Same price and if you lose one
> you just lose 1/10th of your jobs.
>
> Luck!
>
>
>
>
>
> --
> *From*: "haosdent" 
> *Sent*: Saturday, December 17, 2016 6:12 PM
> *To*: "user" 
> *Subject*: Re: Mesos on AWS
>
> >  sometimes Mesos agent is launched but master doesn’t show them.
> It sounds like the Master Master could not connect to your Agents. May you
> mind paste your Mesos Master log? Any information show Mesos agents are
> disconnected in it?
>
> On Sat, Dec 17, 2016 at 4:08 AM, Kiril Menshikov 
> wrote:
>>
>> I have my own framework. Sometimes I get TASK_LOST status with message
>> slave lost during health check.
>>
>> Also I found sometimes Mesos agent is launched but master doesn’t show
>> them. From agent I see that it found master and connected. After agent
>> restart it start working.
>>
>> -Kiril
>>
>>
>>
>> On Dec 16, 2016, at 21:58, Zameer Manji  wrote:
>>
>> Hey,
>>
>> Could you detail on what you mean by "delays and health check problems"?
>> Are you using your own framework or an existing one? How are you launching
>> the tasks?
>>
>> Could you share logs from Mesos that show timeouts to ZK?
>>
>> For reference, I operate a large Mesos cluster and I have never
>> encountered problems when running 1k tasks concurrently so I think sharing
>> data would help everyone debug this problem.
>>
>> On Fri, Dec 16, 2016 at 6:05 AM, Kiril Menshikov 
>> wrote:
>>>
>>> ?Hi,
>>>
>>> Does any body try to run Mesos on AWS instances? Can you give me
>>> recommendations.
>>>
>>> I am developing elastic (scale aws instances on demand) Mesos cluster.
>>> Currently I have 3 master instances. I run about 1000 tasks simultaneously.
>>> I see delays and health check problems.
>>>
>>> ~400 tasks fits in one m4.10xlarge instance. (160GB RAM, 40 CPU).
>>>
>>> At the moment I increase time out in ZooKeeper cluster. What can I do to
>>> decrease timeouts?
>>>
>>> Also how can I increase performance? The main bottleneck is what I have
>>> the big amount of tasks(run simultaneously) for an hour after I shutdown
>>> them or restart (depends how good them perform).
>>>
>>> -Kiril?
>>>
>>> --
>>> Zameer Manji
>>>
>>
>
> --
> Best Regards,
> Haosdent Huang
>



-- 
Thanks,
-Kiril
Phone +37126409291
Riga, Latvia
Skype perimetr122

Re: Structured logging for Mesos (or c++ glog)

[Otis Gospodnetić]

Hi Zhitao,

When people talk about structure and logging it typically means two things:

1) make the log format a known/standard format where all its elements are
known, and thus it's easy to parse them; a log event can still be a single
line, but it can also be multi-line or JSON or some other (even binary)
format.  As long as the format/structure is known, the log event *is*
structured.

2) I want tools/configs/patterns that will let me easily parse this log
event structure and send it somewhere (e.g. Elasticsearch or Logsene
 or ...) where this structure will be handled
in the way that lets me easy filtering/slicing and dicing by one or more
attributes/fields extracted from the log event structure.

*For 1*):
I'm assuming Mesos logs already are structured.  I assume their format is
either widely known (like Apache common log format, for example), or
well-documented (again like Apache common log format).  If that is not
true, then yes, Mesos devs will want to do document the structure.  I've
looked at https://mesos.apache.org/documentation/latest/logging/ but saw
nothing mentioning the structure.  Maybe this info is somewhere else?

*For 2)*
This is where modern log shippers come in. We open-sourced our Logagent
 (more info here
), which has log parsing (and thus
structuring) built-in.  It ships with a bunch of log patterns/parsers, and
one can add new ones (e.g. for Mesos).  Elasticsearch, mentioned in this
thread, is one of the outputs.  It's sort of like Filebeat+Logstash in one,
and it's often used in Dockerized deployments, as part of this Docker agent
.  One could also use Logstash for
parsing/structuring, but Logstash is a bit heavy.

I hope this helps.

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Mon, Dec 19, 2016 at 6:03 PM, Zhitao Li  wrote:

> Charles,
>
> Thanks for sharing the pattern. If my reading is right, this will extract
> the entire message line as one string. What I'm looking for is: on top of
> extracting the entire message line, also break it into structured fields
> automatically.
>
>
>
> On Mon, Dec 19, 2016 at 1:59 PM, Charles Allen <
> charles.al...@metamarkets.com> wrote:
>
>> For what its worth we use SumoLogic and the magic parsing search looks
>> like
>> this:
>>
>> parse regex field=message "^(?[IWE])(?[0-9]{4}
>> [0-9:.]*) [0-9]*
>> (?[0-9a-zA-Z.]*):(?[0-9]*)]
>> (?.*)$"
>>
>>
>>
>> On Mon, Dec 19, 2016 at 11:15 AM Joris Van Remoortere <
>> jo...@mesosphere.io>
>> wrote:
>>
>> > @Zhitao are you looking specifically for structure or just for tagging?
>> > glog does already have support for custom tags in the header. I don't
>> know
>> > if this is enough for your use case though.
>> >
>> > —
>> > *Joris Van Remoortere*
>> > Mesosphere
>> >
>> > On Mon, Dec 19, 2016 at 9:58 AM, James Peach  wrote:
>> >
>> >
>> > > On Dec 19, 2016, at 9:43 AM, Zhitao Li  wrote:
>> > >
>> > > Hi,
>> > >
>> > > I'm looking at how to better utilize ElasticSearch to perform log
>> > analysis for logs from Mesos. It seems like ElasticSearch would
>> generally
>> > work better for structured logging, but Mesos still uses glog thus all
>> logs
>> > produced are old-school unstructured lines.
>> > >
>> > > I wonder whether anyone has brought the conversation of making Mesos
>> > logs easier to process, or if anyone has experience to share.
>> >
>> > Are you trying to stitch together sequences of events? I that case,
>> would
>> > direct event logging be more useful?
>> >
>> > J
>> >
>> >
>> >
>>
>
>
>
> --
> Cheers,
>
> Zhitao Li
>

Re: Welcome Haosdent Huang as Mesos Committer and PMC member!

[Gastón Kleiman]

Congratulations! Very well deserved =).

On Tue, Dec 20, 2016 at 11:17 AM, Aaron Carey  wrote:

> Congratulations! Very well deserved! Always so helpful :)
>
> Aaron Carey
> Production Engineer - Cloud Pipeline
> Industrial Light & Magic
> London
> 020 3751 9150
>
>
> On 20 December 2016 at 01:24, Jie Yu  wrote:
>
>> Congrats! Well deserved!!
>>
>> Always wondering why you have so much time!
>>
>> - Jie
>>
>> On Mon, Dec 19, 2016 at 5:19 PM, Jay Guo 
>> wrote:
>>
>>> Congratulations Haosdent!!!
>>>
>>> /J
>>>
>>> On Mon, Dec 19, 2016 at 4:40 PM, Chengwei Yang
>>>  wrote:
>>> > Congratulations! Well deserved.
>>> >
>>> > Haosdent helps me a lot!
>>> >
>>> > On Fri, Dec 16, 2016 at 01:59:19PM -0500, Vinod Kone wrote:
>>> >> Hi folks,
>>> >>
>>> >> Please join me in formally welcoming Haosdent Huang as Mesos
>>> Committer and
>>> >> PMC member.
>>> >>
>>> >> Haosdent has been an active contributor to the project for more than
>>> a year
>>> >> now. He has contributed a number of patches and features to the Mesos
>>> code
>>> >> base, most notably the unified cgroups isolator and health check
>>> >> improvements. The most impressive thing about him is that he always
>>> >> volunteers to help out people in the community, be it on slack/IRC or
>>> >> mailing lists. The fact that he does all this even though working on
>>> Mesos
>>> >> is not part of his day job is even more impressive.
>>> >>
>>> >> Here is his more formal checklist
>>> >> >> ZGLXG6CF9VP2IY_UU5_0/edit?ts=57e0029d>
>>> >> for your perusal.
>>> >>
>>> >> Thanks,
>>> >> Vinod
>>> >>
>>> >> P.S: Sorry for the delay in sending the welcome email.
>>> >
>>> > --
>>> > Thanks,
>>> > Chengwei
>>>
>>
>>
>

Could mesos be a replacement for yarn?

[Dima Fadeyev]

Hello, everyone,

Is it possible to run software from hadoop ecosystem on mesos? Do these
work: Hive, Oozie, Sqoop, MRv2, Pig?

Thanks in advance and best regards

Re: Structured logging for Mesos (or c++ glog)

[Zhitao Li]

Hi Otis,

Thanks for the good summary. The conversation is mostly about 1) in this
thread, because right now Mesos logs are not really structured, or at least
most of it.

On Tue, Dec 20, 2016 at 6:57 AM, Otis Gospodnetić <
otis.gospodne...@gmail.com> wrote:

> Hi Zhitao,
>
> When people talk about structure and logging it typically means two things:
>
> 1) make the log format a known/standard format where all its elements are
> known, and thus it's easy to parse them; a log event can still be a single
> line, but it can also be multi-line or JSON or some other (even binary)
> format.  As long as the format/structure is known, the log event *is*
> structured.
>
> 2) I want tools/configs/patterns that will let me easily parse this log
> event structure and send it somewhere (e.g. Elasticsearch or Logsene
>  or ...) where this structure will be
> handled in the way that lets me easy filtering/slicing and dicing by one or
> more attributes/fields extracted from the log event structure.
>
> *For 1*):
> I'm assuming Mesos logs already are structured.  I assume their format is
> either widely known (like Apache common log format, for example), or
> well-documented (again like Apache common log format).  If that is not
> true, then yes, Mesos devs will want to do document the structure.  I've
> looked at https://mesos.apache.org/documentation/latest/logging/ but saw
> nothing mentioning the structure.  Maybe this info is somewhere else?
>
> *For 2)*
> This is where modern log shippers come in. We open-sourced our Logagent
>  (more info here
> ), which has log parsing (and thus
> structuring) built-in.  It ships with a bunch of log patterns/parsers, and
> one can add new ones (e.g. for Mesos).  Elasticsearch, mentioned in this
> thread, is one of the outputs.  It's sort of like Filebeat+Logstash in one,
> and it's often used in Dockerized deployments, as part of this Docker
> agent .  One could also use Logstash for
> parsing/structuring, but Logstash is a bit heavy.
>
> I hope this helps.
>
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
> On Mon, Dec 19, 2016 at 6:03 PM, Zhitao Li  wrote:
>
>> Charles,
>>
>> Thanks for sharing the pattern. If my reading is right, this will extract
>> the entire message line as one string. What I'm looking for is: on top of
>> extracting the entire message line, also break it into structured fields
>> automatically.
>>
>>
>>
>> On Mon, Dec 19, 2016 at 1:59 PM, Charles Allen <
>> charles.al...@metamarkets.com> wrote:
>>
>>> For what its worth we use SumoLogic and the magic parsing search looks
>>> like
>>> this:
>>>
>>> parse regex field=message "^(?[IWE])(?[0-9]{4}
>>> [0-9:.]*) [0-9]*
>>> (?[0-9a-zA-Z.]*):(?[0-9]*)]
>>> (?.*)$"
>>>
>>>
>>>
>>> On Mon, Dec 19, 2016 at 11:15 AM Joris Van Remoortere <
>>> jo...@mesosphere.io>
>>> wrote:
>>>
>>> > @Zhitao are you looking specifically for structure or just for tagging?
>>> > glog does already have support for custom tags in the header. I don't
>>> know
>>> > if this is enough for your use case though.
>>> >
>>> > —
>>> > *Joris Van Remoortere*
>>> > Mesosphere
>>> >
>>> > On Mon, Dec 19, 2016 at 9:58 AM, James Peach  wrote:
>>> >
>>> >
>>> > > On Dec 19, 2016, at 9:43 AM, Zhitao Li 
>>> wrote:
>>> > >
>>> > > Hi,
>>> > >
>>> > > I'm looking at how to better utilize ElasticSearch to perform log
>>> > analysis for logs from Mesos. It seems like ElasticSearch would
>>> generally
>>> > work better for structured logging, but Mesos still uses glog thus all
>>> logs
>>> > produced are old-school unstructured lines.
>>> > >
>>> > > I wonder whether anyone has brought the conversation of making Mesos
>>> > logs easier to process, or if anyone has experience to share.
>>> >
>>> > Are you trying to stitch together sequences of events? I that case,
>>> would
>>> > direct event logging be more useful?
>>> >
>>> > J
>>> >
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Cheers,
>>
>> Zhitao Li
>>
>
>


-- 
Cheers,

Zhitao Li