Re: [MESOS-6240] Allow executor/agent communication over non-TCP/IP stream socket.

> what reason for executors need communication with agent Executors need to report task statuses to the agent. Agent needs to send launch task command to the executors. Suppose executors and agents locate in different network namespaces, they could not communicate with each other unless we support

Re: Metrics collection affected when libprocess queue builds up

Hi, @zhitao > the `/metrics/snapshot` could take 10-30 seconds to respond. Do you mean it `/metrics/snapshot` return result after 10~30 seconds? Or `/metrics/snapshot` takes 10~30 seconds to reflect the change of ` allocator/mesos/event_queue_dispatches gauge`? On Mon, Dec 19, 2016 at 1:11 PM,

Re: Map support in proto2

+1 for upgrading proto2 2016-12-19 7:03 GMT+08:00 Neil Conway : > I believe `oneof` is supported in protobuf 2.6.1 [1], so we wouldn't > need to upgrade to make use of it. But I agree that upgrading to > protobuf 3 (while continuing to use the proto2 language version) is >

Re: [MESOS-6240] Allow executor/agent communication over non-TCP/IP stream socket.

haosdent, do you konw unified containerized howto handle the process. 2016-12-20 10:08 GMT+08:00 haosdent : > > what reason for executors need communication with agent > Executors need to report task statuses to the agent. Agent needs to send > launch task command to the

Re: Metrics collection affected when libprocess queue builds up

I believe Zhitao is referring to `/metrics/snapshot` returning a result after 10-30 seconds. I think in a typical environment, this will cause most metrics collection tooling to timeout. This causes the operator to not have any visibility into the system, making debugging/fighting the problem

Re: Mesos Spark Fine Grained Execution - CPU count

Hi Chawla, One possible reason is that Mesos fine grain mode also takes up cores to run the executor per host, so if you have 20 agents running Fine grained executor it will take up 20 cores while it's still running. Tim On Fri, Dec 16, 2016 at 8:41 AM, Chawla,Sumit

Re: Mesos Spark Fine Grained Execution - CPU count

That makes sense. From the documentation it looks like the executors are not supposed to terminate: http://spark.apache.org/docs/latest/running-on-mesos.html#fine-grained-deprecated > Note that while Spark tasks in fine-grained will relinquish cores as they > terminate, they will not relinquish

Structured logging for Mesos (or c++ glog)

Hi, I'm looking at how to better utilize ElasticSearch to perform log analysis for logs from Mesos. It seems like ElasticSearch would generally work better for structured logging, but Mesos still uses glog thus all logs produced are old-school unstructured lines. I wonder whether anyone has

Re: Structured logging for Mesos (or c++ glog)

> On Dec 19, 2016, at 9:43 AM, Zhitao Li wrote: > > Hi, > > I'm looking at how to better utilize ElasticSearch to perform log analysis > for logs from Mesos. It seems like ElasticSearch would generally work better > for structured logging, but Mesos still uses glog

Re: Mesos Spark Fine Grained Execution - CPU count

But coarse grained does the exact same thing which i am trying to avert here. At the cost of lower startup, it keeps the resources reserved till the entire duration of the job. Regards Sumit Chawla On Mon, Dec 19, 2016 at 10:06 AM, Michael Gummelt wrote: > Hi > > I

Re: Welcome Guangya Liu as Mesos Committer and PMC member!

Thank you all! For sure, I'm looking forward to contributing more to the community! We all want to make mesos awesome! On Mon, Dec 19, 2016 at 4:51 PM, tommy xiao wrote: > Congrats Guangya! > > 2016-12-19 11:26 GMT+08:00 Yan Xu : > >> Congrats! >> >> --- >> Jiang

Re: Welcome Guangya Liu as Mesos Committer and PMC member!

Congrats Guangya! 2016-12-19 11:26 GMT+08:00 Yan Xu : > Congrats! > > --- > Jiang Yan Xu | @xujyan > > On Mon, Dec 19, 2016 at 1:31 AM, haosdent wrote: > >> Congrats Guangya! >> >> On Sun, Dec 18, 2016 at 10:02 PM,

Re: Mesos Spark Fine Grained Execution - CPU count

Ah thanks. looks like i skipped reading this *"Neither will executors terminate when they’re idle."* So in my job scenario, I should preassume that No of executors should be less than number of tasks. Ideally one executor should execute 1 or more tasks. But i am observing something strange

Re: thread_local supported on Apple

I believe this thread_local support is in XCode 8.2. From the link you shared: > Xcode 8.2 requires a Mac running macOS 10.11.5 or later This means that users can upgrade the compiler on El Capitan just fine without a system upgrade. Users on Yosemite need to do a system upgrade to pick up the

Re: thread_local supported on Apple

Thanks for your input Zameer. Is it common for developers on mac to use XCode as their compilation environment as well? I would think if you used clang on the command line then you could still install an updated version of clang without having to do a system upgrade from Yosemite? I'm getting

Re: Structured logging for Mesos (or c++ glog)

Charles, Thanks for sharing the pattern. If my reading is right, this will extract the entire message line as one string. What I'm looking for is: on top of extracting the entire message line, also break it into structured fields automatically. On Mon, Dec 19, 2016 at 1:59 PM, Charles Allen <

Re: Mesos Spark Fine Grained Execution - CPU count

We will be interested by the results if you give a try to Dynamic allocation with mesos ! - Mail Original - De: "Michael Gummelt" À: "Sumit Chawla" Cc: u...@mesos.apache.org, dev@mesos.apache.org, "User" ,

Re: Structured logging for Mesos (or c++ glog)

Joris, I am particular looking for structure. We have mechanism to add static tags easily to log collected into ELK. If there is a way to dynamically inject tags like "framework_id" at actual logging call, it might be a starting point for me. I cannot find a good reference on how to add tagging

Re: Mesos Spark Fine Grained Execution - CPU count

Dynamic allocation works with Coarse grain mode only, we wasn't aware a need for Fine grain mode after we enabled dynamic allocation support on the coarse grain mode. What's the reason you're running fine grain mode instead of coarse grain + dynamic allocation? Tim On Mon, Dec 19, 2016 at 2:45

Re: [MESOS-6240] Allow executor/agent communication over non-TCP/IP stream socket.

don't understand what reason for executors need communication with agent? 2016-12-19 19:54 GMT+08:00 pangbingqiang : > Hi all: > >What’s the latest information about MESOS-6240 > https://issues.apache.org/jira/browse/MESOS-6240 ,have any demo or design > achieve? >

Re: Structured logging for Mesos (or c++ glog)

For what its worth we use SumoLogic and the magic parsing search looks like this: parse regex field=message "^(?[IWE])(?[0-9]{4} [0-9:.]*) [0-9]* (?[0-9a-zA-Z.]*):(?[0-9]*)] (?.*)$" On Mon, Dec 19, 2016 at 11:15 AM Joris Van Remoortere wrote: > @Zhitao are you looking

Re: thread_local supported on Apple

Is my understanding incorrect regarding the ability to upgrade the compiler version on Yosemite and El Capitan without requiring a full system upgrade? @Mpark are you making a case for not updating to `thread_local` just yet? — *Joris Van Remoortere* Mesosphere On Fri, Dec 16, 2016 at 11:11 AM,

Re: Structured logging for Mesos (or c++ glog)

@Zhitao are you looking specifically for structure or just for tagging? glog does already have support for custom tags in the header. I don't know if this is enough for your use case though. — *Joris Van Remoortere* Mesosphere On Mon, Dec 19, 2016 at 9:58 AM, James Peach

Re: Welcome Haosdent Huang as Mesos Committer and PMC member!

Congratulations Haosdent!!! /J On Mon, Dec 19, 2016 at 4:40 PM, Chengwei Yang wrote: > Congratulations! Well deserved. > > Haosdent helps me a lot! > > On Fri, Dec 16, 2016 at 01:59:19PM -0500, Vinod Kone wrote: >> Hi folks, >> >> Please join me in formally welcoming

Re: thread_local supported on Apple

> On Dec 19, 2016, at 3:00 PM, Joris Van Remoortere wrote: > > Thanks for your input Zameer. > > Is it common for developers on mac to use XCode as their compilation > environment as well? I would think if you used clang on the command line > then you could still install

Re: Welcome Haosdent Huang as Mesos Committer and PMC member!

Congrats! Well deserved!! Always wondering why you have so much time! - Jie On Mon, Dec 19, 2016 at 5:19 PM, Jay Guo wrote: > Congratulations Haosdent!!! > > /J > > On Mon, Dec 19, 2016 at 4:40 PM, Chengwei Yang > wrote: > >