Re: how to debug when a task is killed

Do you configure health check? If you configure health check and it could not pass, the task would be killed. On Tue, Dec 20, 2016 at 2:23 PM, Luke Adolph wrote: > Hi all: > > I have set up a mesos cluster with on mesos master and five mesos agents. > I use Marathon to depoy an app across mesos

how to debug when a task is killed

Hi all: I have set up a mesos cluster with on mesos master and five mesos agents. I use Marathon to depoy an app across mesos agents, which reads process info from /proc. About every 40 minuntes, my apps will be killed and Marathon restart them. The stderr info in sandbox is: ​ I1220 05:05:12.014

Re: [MESOS-6240] Allow executor/agent communication over non-TCP/IP stream socket.

haosdent, do you konw unified containerized howto handle the process. 2016-12-20 10:08 GMT+08:00 haosdent : > > what reason for executors need communication with agent > Executors need to report task statuses to the agent. Agent needs to send > launch task command to the executors. > Suppose exec

Re: Mesos 1.1 web ui issues

Hi, @haripriya Ping me in Mesos Slack (https://mesos.slack.com/) when you are available, I think it would speed up the progress to solve your problem. My id is @haosdent. If you have not join Mesos Slack before, you could join it via https://mesos-slackin.herokuapp.com . On Tue, Dec 20, 2016 at 2:

Re: Libraries to access Mesos HTTP endpoints

As I know, don't have libraries for v1 operator APIs so far. > DC/OS CLI [2] seems including those features, but it's too much and not programmable for me (unless I try parsing its output). dcos cli is open source as well. You may refer its implementation. https://github.com/dcos/dcos-cli/blob/mas

Re: [MESOS-6240] Allow executor/agent communication over non-TCP/IP stream socket.

> what reason for executors need communication with agent Executors need to report task statuses to the agent. Agent needs to send launch task command to the executors. Suppose executors and agents locate in different network namespaces, they could not communicate with each other unless we support

Re: Structured logging for Mesos (or c++ glog)

Why not `logstash`? I think it's the target case of `logstash`. On Tue, Dec 20, 2016 at 7:35 AM Zhitao Li wrote: > Great. > > I also found this old thread > http://search-hadoop.com/m/Mesos/0Vlr6meKs116T2k1?subj=Mapped+diagnostics+context+Adding+internal+Mesos+IDs+as+context+to+the+logs > on >

Re: Welcome Haosdent Huang as Mesos Committer and PMC member!

Congrats! Well deserved!! Always wondering why you have so much time! - Jie On Mon, Dec 19, 2016 at 5:19 PM, Jay Guo wrote: > Congratulations Haosdent!!! > > /J > > On Mon, Dec 19, 2016 at 4:40 PM, Chengwei Yang > wrote: > > Congratulations! Well deserved. > > > > Haosdent helps me a lot! > >

Re: Structured logging for Mesos (or c++ glog)

Great. I also found this old thread http://search-hadoop.com/m/Mesos/0Vlr6meKs116T2k1?subj=Mapped+diagnostics+context+Adding+internal+Mesos+IDs+as+context+to+the+logs on dev list, which seems no consensus has been made. Maybe we can talk about this in the next community sync? On Mon, Dec 19, 201

Re: Structured logging for Mesos (or c++ glog)

> On Dec 19, 2016, at 2:54 PM, Zhitao Li wrote: > > Hi James, > > Stitching events together is only one possible use cases, and I'm not exactly > sure what you meant by directly event logging. > > Taking the hierarchical allocator for example. In a multi-framework cluster, > sometimes I want

Re: Mesos Spark Fine Grained Execution - CPU count

Tim, We will try to run the application in coarse grain mode, and share the findings with you. Regards Sumit Chawla On Mon, Dec 19, 2016 at 3:11 PM, Timothy Chen wrote: > Dynamic allocation works with Coarse grain mode only, we wasn't aware > a need for Fine grain mode after we enabled dynami

Re: [MESOS-6240] Allow executor/agent communication over non-TCP/IP stream socket.

don't understand what reason for executors need communication with agent? 2016-12-19 19:54 GMT+08:00 pangbingqiang : > Hi all: > >What’s the latest information about MESOS-6240 > https://issues.apache.org/jira/browse/MESOS-6240 ,have any demo or design > achieve? > > I see libprocess have su

Re: Mesos Spark Fine Grained Execution - CPU count

Dynamic allocation works with Coarse grain mode only, we wasn't aware a need for Fine grain mode after we enabled dynamic allocation support on the coarse grain mode. What's the reason you're running fine grain mode instead of coarse grain + dynamic allocation? Tim On Mon, Dec 19, 2016 at 2:45 P

Re: Structured logging for Mesos (or c++ glog)

Charles, Thanks for sharing the pattern. If my reading is right, this will extract the entire message line as one string. What I'm looking for is: on top of extracting the entire message line, also break it into structured fields automatically. On Mon, Dec 19, 2016 at 1:59 PM, Charles Allen < c

Re: Structured logging for Mesos (or c++ glog)

Joris, I am particular looking for structure. We have mechanism to add static tags easily to log collected into ELK. If there is a way to dynamically inject tags like "framework_id" at actual logging call, it might be a starting point for me. I cannot find a good reference on how to add tagging

Re: Structured logging for Mesos (or c++ glog)

Hi James, Stitching events together is only one possible use cases, and I'm not exactly sure what you meant by directly event logging. Taking the hierarchical allocator for example. In a multi-framework cluster, sometimes I want to comb through various loggings and present a trace on how allocati

Re: Structured logging for Mesos (or c++ glog)

For what its worth we use SumoLogic and the magic parsing search looks like this: parse regex field=message "^(?[IWE])(?[0-9]{4} [0-9:.]*) [0-9]* (?[0-9a-zA-Z.]*):(?[0-9]*)] (?.*)$" On Mon, Dec 19, 2016 at 11:15 AM Joris Van Remoortere wrote: > @Zhitao are you looking specifically for structu

Re: Mesos Spark Fine Grained Execution - CPU count

> Is this problem of idle executors sticking around solved in Dynamic Resource Allocation? Is there some timeout after which Idle executors can just shutdown and cleanup its resources. Yes, that's exactly what dynamic allocation does. But again I have no idea what the state of dynamic allocation

Re: Mesos Spark Fine Grained Execution - CPU count

Great. Makes much better sense now. What will be reason to have spark.mesos.mesosExecutor.cores more than 1, as this number doesn't include the number of cores for tasks. So in my case it seems like 30 CPUs are allocated to executors. And there are 48 tasks so 48 + 30 = 78 CPUs. And i am noti

Re: Mesos Spark Fine Grained Execution - CPU count

> I should preassume that No of executors should be less than number of tasks. No. Each executor runs 0 or more tasks. Each executor consumes 1 CPU, and each task running on that executor consumes another CPU. You can customize this via spark.mesos.mesosExecutor.cores ( https://github.com/apac

Re: Mesos Spark Fine Grained Execution - CPU count

Ah thanks. looks like i skipped reading this *"Neither will executors terminate when they’re idle."* So in my job scenario, I should preassume that No of executors should be less than number of tasks. Ideally one executor should execute 1 or more tasks. But i am observing something strange inste

Re: Mesos Spark Fine Grained Execution - CPU count

That makes sense. From the documentation it looks like the executors are not supposed to terminate: http://spark.apache.org/docs/latest/running-on-mesos.html#fine-grained-deprecated > Note that while Spark tasks in fine-grained will relinquish cores as they > terminate, they will not relinquish me

Re: Mesos Spark Fine Grained Execution - CPU count

Hi Chawla, One possible reason is that Mesos fine grain mode also takes up cores to run the executor per host, so if you have 20 agents running Fine grained executor it will take up 20 cores while it's still running. Tim On Fri, Dec 16, 2016 at 8:41 AM, Chawla,Sumit wrote: > Hi > > I am using S

Re: Mesos Spark Fine Grained Execution - CPU count

Yea, the idea is to use dynamic allocation. I can't speak to how well it works with Mesos, though. On Mon, Dec 19, 2016 at 11:01 AM, Mehdi Meziane wrote: > I think that what you are looking for is Dynamic resource allocation: > http://spark.apache.org/docs/latest/job-scheduling.html# > dynamic-

Re: Structured logging for Mesos (or c++ glog)

@Zhitao are you looking specifically for structure or just for tagging? glog does already have support for custom tags in the header. I don't know if this is enough for your use case though. — *Joris Van Remoortere* Mesosphere On Mon, Dec 19, 2016 at 9:58 AM, James Peach wrote: > > > On Dec 19,

Re: Mesos Spark Fine Grained Execution - CPU count

But coarse grained does the exact same thing which i am trying to avert here. At the cost of lower startup, it keeps the resources reserved till the entire duration of the job. Regards Sumit Chawla On Mon, Dec 19, 2016 at 10:06 AM, Michael Gummelt wrote: > Hi > > I don't have a lot of experie

Re: Mesos 1.1 web ui issues

Hi @Haosdent, We have multiple networks- that could be one of the problems. I tried with all 3 of them and it still shows the same error. Can you help me understand what hostname exactly expects in such scenario? On Thu, Dec 15, 2016 at 6:08 PM, haosdent wrote: > Hi, @haripriya What's the hostn

Re: Mesos Spark Fine Grained Execution - CPU count

Hi I don't have a lot of experience with the fine-grained scheduler. It's deprecated and fairly old now. CPUs should be relinquished as tasks complete, so I'm not sure why you're seeing what you're seeing. There have been a few discussions on the spark list regarding deprecating the fine-graine

Re: Structured logging for Mesos (or c++ glog)

> On Dec 19, 2016, at 9:43 AM, Zhitao Li wrote: > > Hi, > > I'm looking at how to better utilize ElasticSearch to perform log analysis > for logs from Mesos. It seems like ElasticSearch would generally work better > for structured logging, but Mesos still uses glog thus all logs produced are

Structured logging for Mesos (or c++ glog)

Hi, I'm looking at how to better utilize ElasticSearch to perform log analysis for logs from Mesos. It seems like ElasticSearch would generally work better for structured logging, but Mesos still uses glog thus all logs produced are old-school unstructured lines. I wonder whether anyone has broug

Re: Welcome Guangya Liu as Mesos Committer and PMC member!

Thank you all! For sure, I'm looking forward to contributing more to the community! We all want to make mesos awesome! On Mon, Dec 19, 2016 at 4:51 PM, tommy xiao wrote: > Congrats Guangya! > > 2016-12-19 11:26 GMT+08:00 Yan Xu : > >> Congrats! >> >> --- >> Jiang Yan Xu | @xujyan

Re: Welcome Guangya Liu as Mesos Committer and PMC member!

Congrats Guangya! 2016-12-19 11:26 GMT+08:00 Yan Xu : > Congrats! > > --- > Jiang Yan Xu | @xujyan > > On Mon, Dec 19, 2016 at 1:31 AM, haosdent wrote: > >> Congrats Guangya! >> >> On Sun, Dec 18, 2016 at 10:02 PM, Klaus Ma >> wrote: >> >>> Congratulations!! >>> >>

Re: Welcome Haosdent Huang as Mesos Committer and PMC member!

Congratulations! Well deserved. Haosdent helps me a lot! On Fri, Dec 16, 2016 at 01:59:19PM -0500, Vinod Kone wrote: > Hi folks, > > Please join me in formally welcoming Haosdent Huang as Mesos Committer and > PMC member. > > Haosdent has been an active contributor to the project for more than