Re: How to send a task to a running framework?

[Vinod Kone]

Are you asking how users can submit their tasks to your custom framework?
Your framework should probably expose an API for that.

On Fri, Jul 8, 2016 at 4:30 AM, Bryan Fok  wrote:

> Hi all
>
> After I have my custom framework running in , for instance, a python
> process
>
> How do I submit a task to it through another python process from another
> machine? Through the framework name? Any document around this?
>
> BR
> Bryan
>

Re: Return a vector value from custom executor

[Bryan Fok]

I end up using a shared folder to allow both parties to share resources
both input and output

On Wed, Jul 6, 2016 at 10:37 AM, Bryan Fok  wrote:

> Understood, moving to that direction now. Thank you for your advance.
>
> BR
> Bryan
>
> On Wed, Jul 6, 2016 at 1:29 AM, haosdent  wrote:
>
>> I afraid protobuf don't allow so large message size? I think you may
>> store your result in HDFS or other distributed filesystem. And executor
>> just notify framework the progress of store.
>>
>> On Tue, Jul 5, 2016 at 2:17 PM, Bryan Fok 
>> wrote:
>>
>>> Hi all
>>>
>>> What is the best practice to have a custom executor to return large size
>>> (~200mb) value (e.g. matrix array, or numpy array) ?
>>>
>>> B.R
>>> Bryan
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>

Re: Mesos API - How to send argument to task?

[Bryan Fok]

I end up using a shared folder, so both parties are using the same path to
referencing resources

On Tue, Jul 5, 2016 at 2:48 PM, Bryan Fok  wrote:

> Thanks bro, thanks for your advance. I will try it out and return feedback
> here
>
> On Tue, Jul 5, 2016 at 2:23 PM, Gilbert Song 
> wrote:
>
>> Bryan,
>>
>> You can define your own arguments in your
>> 'TaskInfo::CommandInfo::arguments',
>> while having your executable as 'value'. Please note that 'shell'
>> defaults to be true
>> in CommandInfo.
>>
>> Gilbert
>>
>> On Mon, Jul 4, 2016 at 11:11 PM, Bryan Fok 
>> wrote:
>>
>>>
>>>
>>>
>>>
>>>
>>> 
>>> Hi all
>>>
>>> I am writing a python framework which has a custom executor. Each task I
>>> submit will need to pass 2 string as arguments, each task also need 2
>>> unique xml configuration files in the slave for the custom executor. So how
>>> do I pass arguments around for each task, as well as files as arguments?
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

How to send a task to a running framework?

[Bryan Fok]

Hi all

After I have my custom framework running in , for instance, a python process

How do I submit a task to it through another python process from another
machine? Through the framework name? Any document around this?

BR
Bryan

Re: Setting up SSL for mesos

[Kamil Wokitajtis]

You don't have to build it on every machine.
Just install checkinstall tool, build it on one machine and create package.
When building, after make command don't run make install, instead run:

checkinstall --install=no --pkgname=mesos --pkgversion=0.28.2 -y

I am using Ubuntu, so in my case it will create deb package.
Just copy it over to every node and do:

dpkg -i 

With this approach, it is very easy to update mesos to the most recent
version.


2016-07-08 1:56 GMT+02:00 Douglas Nelson :

> Inlining the environment variable fixed it! I had to use sudo to run
> mesos, so I either needed to inline it or use -E. *Facepalm*
>
> On the note of building mesos, is there any chance in the future that
> mesosphere would create a prebuilt mesos package with ssl included? It
> would be nice to avoid building on each machine I want to run mesos on.
> Also, it would be great to have the convenience of using config files,
> having automatic startup, etc with a simple apt-get install mesos-ssl. Is
> that something that's been considered?
>
> Thanks!
>
> On Thu, Jul 7, 2016 at 1:14 PM, Joseph Wu  wrote:
>
>> Can you double-check if your master is linking to openssl?
>>
>> From your build folder, you should get something like:
>> ldd src/.libs/mesos-master | grep ssl
>> libevent_openssl-2.0.so.5 => /lib64/libevent_openssl-2.0.so.5
>> libssl.so.10 => /lib64/libssl.so.10
>> libssl3.so => /lib64/libssl3.so
>>
>> There doesn't seem to be anything wrong with your configure/build steps.
>> And your environment variables setup should work on any sane Unix shell.
>> (Perhaps inline the environment variable?  SSL_ENABLED=true
>> ./mesos-master.sh ...)
>>
>> On Thu, Jul 7, 2016 at 11:53 AM, Douglas Nelson 
>> wrote:
>>
>>> I rebuilt from scratch with SSL support and got no errors. I only set 
>>> *export
>>> SSL_ENABLED=true* and then I ran the mesos-master.
>>>
>>> No errors were thrown and I can see the web UI via HTTP. I double
>>> checked that I was running the .sh from the build folder I created. Is
>>> mesos not connecting with the environment variable I set for some reason?
>>>
>>>
>>>
>>> On Wed, Jul 6, 2016 at 2:20 PM, Joseph Wu  wrote:
>>>
 If you can see the WebUI via HTTP, without downgrade support, you might
 be inadvertently running a different version of Mesos than the one you
 built.

 You can quickly sanity check this by removing either SSL_KEY_FILE or
 SSL_CERT_FILE and starting your master.  If your build has SSL support, it
 should immediately exit with an error message.


 On Wed, Jul 6, 2016 at 12:33 PM, Douglas Nelson 
 wrote:
 >
 > I attempted to set up SSL following this guide:
 http://mesos.apache.org/documentation/latest/ssl/
 >
 > I'm able to hit the WebUI with http but using https gives me nothing.
 I must be missing something. Here are the steps I'm taking:
 >
 > I downloaded 0.28.2 from here:
 https://github.com/apache/mesos/releases
 > I ran ./configure --enable-libevent --enable-ssl
 > Then I ran make and make install
 > I set the following environment variables:
 >
 > export SSL_ENABLED=1
 > export SSL_SUPPORT_DOWNGRADE=0
 > export SSL_KEY_FILE=
 > export SSL_CERT_FILE=
 >
 > Finally, I ran ./bin/mesos-master.sh --ip=127.0.0.1
 --work_dir=/var/lib/mesos
 >
 > I can provide any additional information if needed. Thanks!
 >
 > Also, I read that SSL would be included in mesosphere's nightly
 builds: https://open.mesosphere.com/downloads/mesos-nightly/
 >
 > How stable are those builds and has SSL already been included?
 >
 > -Doug Nelson


>>>
>>
>

Re: Monitoring at container level

[Michał Łowicki]

For now the easiest solution not requiring changing monitoring
infrastructure would be the one proposed by Steven Schlansker so will try
to get some information about it from Marathon team. Thanks!

On Fri, Jul 8, 2016 at 5:20 AM,  wrote:

> Small plug for snap (https://github.com/intelsdi-x/snap). It's a
> telemetry framework with a lot of useful plugins for collecting, processing
> and publishing metrics. There's a go API (and soon more langs) for writing
> your own plugins. Plugin catalog:
> https://github.com/intelsdi-x/snap/blob/master/docs/PLUGIN_CATALOG.md
>
> On Jul 7, 2016, at 17:34, Guangya Liu  wrote:
>
> Have you ever tried prometheus + Grafana? Please take a look at
> https://prometheus.io/docs/visualization/grafana/ to see if it helps.
>
> On Fri, Jul 8, 2016 at 5:51 AM, David Kesler  wrote:
>
>> We use a combination of new relic for application level monitoring and a
>> custom python script that scrapes a bunch of stats from the docker socket
>> file and throws them into elastic so we can use kibana to make graphs.
>>
>>
>>
>> *From:* Gregory Durham [mailto:gregory.dur...@gmail.com]
>> *Sent:* Thursday, July 07, 2016 4:58 PM
>> *To:* user@mesos.apache.org
>> *Cc:* krishnan.k.i...@gmail.com; Michał Łowicki
>> *Subject:* Re: Monitoring at container level
>>
>>
>>
>> I have been using datadog to monitor my infrastructure. The integration
>> into service discovery has been really helpful for these environments.
>>
>>
>>
>> On Thu, Jul 7, 2016 at 1:37 PM, Steven Schlansker <
>> sschlans...@opentable.com> wrote:
>>
>> We use Graphite and ran into similar problems with huge metric namespaces.
>> We use the Singularity framework which provides both the task "request
>> id" (name)
>> and "instance number" (0..N) to the task.
>>
>> So we set our Graphite namespace to be "request-number" e.g. "myservice-3"
>> This has the downside of discontinuous data when you deploy a new release
>> but we haven't had too many issues due to that in practice.
>>
>>
>>
>> > On Jul 7, 2016, at 1:26 PM, Krish  wrote:
>> >
>> > I have had a good experience so far with bosun and scollector with
>> cadvisor.
>> > Check it out at bosun.org.
>> >
>> >
>> > On Friday 8 July 2016, Pradeep Chhetri 
>> wrote:
>> > Hi Michal,
>> >
>> > Do have a look at sysdig (http://www.sysdig.org). It is basically an
>> open-source tool which provides container insights. Maybe your will find
>> something helpful over there.
>> >
>> > To tackle the case of new metrics for new containers, maybe you should
>> tag metrics by service-name instead of container id. (Graphite doesn't have
>> concept of tags but something like opentsdb and influxdb do have. I don't
>> see a reason to replace graphite for that. You can use your service-name
>> (which the container is representing) instead of hostname in the metrics
>> name)
>> >
>> > On Fri, Jul 8, 2016 at 1:18 AM, Michał Łowicki 
>> wrote:
>> > Hi,
>> >
>> > Before introducing Mesos we're using mainly Graphite / Grafana. Ideally
>> we would like to have metrics per container as an easy way to detect if
>> problem touches only single, subset of containers or it's global.
>> >
>> > Unfortunately using Graphite for that is far from being perfect. Having
>> container identifier as a part of metric has many negative implications
>> like having tons of new metrics every release on Marathon (new containers =
>> new identifiers).
>> >
>> > Investigated InfluxDB so far but project isn't mature enough as still
>> components like
>> https://github.com/influxdata/telegraf/blob/master/plugins/inputs/statsd/README.md#influx-statsd
>> have major blockers:
>> >
>> > COMING SOON: there will be a way to specify multiple fields.
>> >
>> > What do you use to monitor your Mesos clusters and f.ex. to detect that
>> some containers are having issues?
>> >
>> > --
>> > BR,
>> > Michał Łowicki
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Pradeep Chhetri
>> >
>> >
>> > --
>> >
>> > Thumb typed mail
>> >
>>
>>
>>
>
>


-- 
BR,
Michał Łowicki