Re: spark on mesos with docker issue

2015-10-22 Thread Rad Gruchalski
on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately. On Thursday, 22 October 2015 at 22:28, Iulian Dragoș wrote: > > > On Thu, Oct 22, 2015 at 9:10 PM, Rad Gruchalski <ra...@gruchalski.com > (mailto:ra...@gruchal

Re: Tasks that run docker images consistently fail while downloading

2015-10-28 Thread Rad Gruchalski
but my > tasks are killed off after 1 minute without being allowed to fully download > the image. > > Jim > > On 28 October 2015 at 10:26, Rad Gruchalski <ra...@gruchalski.com > (mailto:ra...@gruchalski.com)> wrote: > > Jim, > > > > Ha

Re: Tasks that run docker images consistently fail while downloading

2015-10-28 Thread Rad Gruchalski
/destroy and inform the sender immediately. On Wednesday, 28 October 2015 at 13:08, Rad Gruchalski wrote: > Nobody getting those today ;) Good catch. Worth keeping in mind! > > > > > > > > > > > Kind regards,
 > Radek Grucha

Re: spark on mesos with docker issue

2015-11-10 Thread Rad Gruchalski
nce right? > > On Thu, Oct 22, 2015 at 10:38 PM, Rad Gruchalski <ra...@gruchalski.com > (mailto:ra...@gruchalski.com)> wrote: > > There are 2 things: > > > > - Akka remote in 2.3.x does not support advertising hostname / port > > different to what it

Re: How is Mesos doing certificate verification for resources in URIs?

2015-11-04 Thread Rad Gruchalski
for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately. On Wednesday, 4 November 2015 at 12:31, Rad Gruchalski wrote

How is Mesos doing certificate verification for resources in URIs?

2015-11-04 Thread Rad Gruchalski
Hi everyone, I’ve added the following URI to the URIs for the task: https://raw.githubusercontent.com/apache/spark/master/pom.xml. However, my task has failed because of: Failed to fetch 'https://raw.githubusercontent.com/apache/spark/master/pom.xml': Error downloading resource: Peer

Re: How is Mesos doing certificate verification for resources in URIs?

2015-11-04 Thread Rad Gruchalski
Yes, this is from the agent: ~$ curl -i https://raw.githubusercontent.com/apache/spark/master/pom.xml HTTP/1.1 200 OK Content-Security-Policy: default-src 'none' X-XSS-Protection: 1; mode=block X-Frame-Options: deny X-Content-Type-Options: nosniff Strict-Transport-Security: max-age=31536000

Re:

2015-10-11 Thread Rad Gruchalski
Rafael, According to the cassandra documentation, you should not be affected at all: http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_tune_jvm_c.html However, your performance with these settings will be rather poor. Kind regards,
 Radek Gruchalski 


Re: Old docker version deployed

2015-10-06 Thread Rad Gruchalski
Paul, Are you using the same tag every time? Kind regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This communication is intended

Re: Mesos 0.24.0 with spark in docker - error

2015-09-16 Thread Rad Gruchalski
sr/lib/libmesos-0.24.0.so > > (http://libmesos-0.24.0.so/)) > > You are right: it's clearly a missing dependency in libcurl (FWIW, I don't > think Mesos supports 12.04). > > Marco Massenzio > Distributed Systems Engineer > http://codetrips.com > On Wed, Sep 16, 2015 a

Mesos 0.24.0 with spark in docker - error

2015-09-16 Thread Rad Gruchalski
Dear list, I’m here for the first time so I do apologise if I’m misbehaving. I am currently trying running Scala Spark Notebook on Mesos with Docker in bridge networking. I would like to launch spark executors using the cluster, the master is running on. This requires registering a framework

Re: mesos-tail in 0.24.1

2015-09-29 Thread Rad Gruchalski
> (mailto:haosd...@gmail.com)> wrote: > > I think you install mesos-cli from https://github.com/mesosphere/mesos-cli > > > > On Tue, Sep 29, 2015 at 8:51 PM, Rad Gruchalski <ra...@gruchalski.com > > (mailto:ra...@gruchalski.com)> wrote: > > > It seems that

Re: mesos-tail in 0.24.1

2015-09-29 Thread Rad Gruchalski
i.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no acti

mesos-tail in 0.24.1

2015-09-29 Thread Rad Gruchalski
Hi everyone, I have upgraded my development mesos environment to 0.24.1 this morning. It’s a clean installation with new zookeeper and everything. Since the upgrade I get an error while executing mesos-tail: mesos-master ~$ mesos tail -f -n 50 service Traceback (most recent call last): File

Re: mesos-tail in 0.24.1

2015-09-30 Thread Rad Gruchalski
data to ZooKeeper. > > > > This is a genuine bug, thanks for reporting: would you mind terribly to > > file a Jira and assign to me, please? > > (marco-mesos) > > > > Thanks! > > > > Marco Massenzio > > Distributed Systems Engineer > &

Re: mesos-tail in 0.24.1

2015-09-30 Thread Rad Gruchalski
would you mind terribly to file > a Jira and assign to me, please? > (marco-mesos) > > Thanks! > > Marco Massenzio > Distributed Systems Engineer > http://codetrips.com > On Tue, Sep 29, 2015 at 6:28 AM, Rad Gruchalski <ra...@gruchalski.com > (mailto:ra...@gr

Increasing a number of ports for a Docker container

2015-10-01 Thread Rad Gruchalski
Hi everyone, I remember reading somewhere in the documentation that there’s a limit of number of port mappings for a single task / container. I believe it is 10, however, I can’t find the specific documentation link describing this. Is it possible to increase this number? Say, I need 15 not

Re: Official RPMs

2015-09-18 Thread Rad Gruchalski
Should be rather easy to package it with this little tool from Mesosphere: https://github.com/mesosphere/mesos-deb-packaging. I’ve done it myself for ubuntu 12.04 and 14.04. The only thing that needs to be changed are the dependencies, for ubuntu this was: diff --git a/build_mesos

Learning Agent ID without reading full state?

2015-12-08 Thread Rad Gruchalski
Hi there, I’m trying to figure out what would the best way to learn the agent id without having to read the complete state of mesos. Currently, I do the following: mesos-state 2>/dev/null | jq '.slaves[] | select(.hostname == "’$(AGENT_HOSTNAME)'") | .id' -r However, depending on how much

Re: Learning Agent ID without reading full state?

2015-12-08 Thread Rad Gruchalski
; please delete/destroy and inform the sender immediately. On Tuesday, 8 December 2015 at 15:42, tommy xiao wrote: > why are you need the Agent ID? could you please input more information? > > 2015-12-08 19:27 GMT+08:00 Rad Gruchalski <ra...@gruchalski.com > (mailto:ra...@gruchalsk

Re: resolving hosts with mesos-dns not working with "/" in the appid

2015-11-18 Thread Rad Gruchalski
According to the docs, it is possible: Mesos-DNS follows RFC 952 for name formatting. All fields used to construct hostnames for A records and service names for SRV records must be up to 24 characters and drawn from the alphabet (A-Z), digits (0-9) and minus sign (-). No distinction is made

Re: Spark task sometimes won't start

2015-11-24 Thread Rad Gruchalski
Mitsutoshi, Is this in a fine-grained mode? Kind regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This communication is intended

Re: Spark task sometimes won't start

2015-11-24 Thread Rad Gruchalski
; please delete/destroy and inform the sender immediately. On Tuesday, 24 November 2015 at 10:07, 木内満歳 wrote: > Hi Rad, > > I've tried both. I've experienced same symptom on both case. > > Thanks, > Mitsutoshi Kiuchi > > > 2015-11-24 17:57 GMT+09:00 Rad Gruch

Re: Spark task sometimes won't start

2015-11-24 Thread Rad Gruchalski
. On Tuesday, 24 November 2015 at 11:32, 木内満歳 wrote: > I agree. "coarse grain mode" is "take me to safe side" mode, I think. > Unfortunately, it looks more hard to resolve... > > Anyway, I appreciate your advice. Thanks much ! > > Mitsutoshi > >

Re: How to deploy a Database cluster

2016-02-24 Thread Rad Gruchalski
Alfredo, Not sure how to do this with Calico and Co but you need to investigate LIBPROCESS_ADVERTISE_IP and LIBPROCESS_ADVERTICE_PORT. https://github.com/apache/mesos/blob/master/docs/configuration.md#libprocess-options Basically, what you need to do is, in your container you need to: export

mesos.interface 0.27.2

2016-03-10 Thread Rad Gruchalski
Hi everyone, I have successfully upgraded my cluster to 0.27.2 but I have a couple of questions regarding mesos.interface python package. First, I do not see the 0.27.2 version in pypi: https://pypi.python.org/pypi/mesos.interface. Is that intentional? Second, I thought it would be better

Re: mess spark cluster mode error

2016-03-14 Thread Rad Gruchalski
I believe you are missing either —spark.executor.uri pointing to spark tar.gz or —spark.mesos.executor.home http://spark.apache.org/docs/latest/running-on-mesos.html#configuration Kind regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)


Re: Apache Spark Over Mesos

2016-03-15 Thread Rad Gruchalski
That’s a great alternative! Kind regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This communication is intended for the

Re: Apache Spark Over Mesos

2016-03-15 Thread Rad Gruchalski
As Tim suggested: spark.mesos.executor.docker.image is your friend. Kind regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This

Re: 0.27.2 RPMs

2016-03-09 Thread Rad Gruchalski
Was it actually released? I can see the version tagged in git but there was no official release note? Kind regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/

Re: 0.27.2 RPMs

2016-03-09 Thread Rad Gruchalski
Oh, cool, I was expecting some info here. Going to build one now :) Kind regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This

Re: Mesos 0.27 and docker

2016-03-11 Thread Rad Gruchalski
) wrote: > You are specifying two ways, what’s the preferred way? > > Walter > > > From: Rad Gruchalski [mailto:ra...@gruchalski.com] > Sent: 11 March 2016 12:20 > To: user@mesos.apache.org (mailto:user@mesos.apache.org) > Subject: Re: Mesos 0.27

Re: marathon issue in running a docker container.

2016-04-12 Thread Rad Gruchalski
Do you have anything like haproxy for port mappings installed on your Mesos cluster? When using BRIDGE network, your process inside of the container, say SSH, is running on a standard port 22. Marathon allocates a random port in the default range of 31000 to 32000. However, it is your task to

Re: orphaned_tasks cleanup and prevention method

2016-04-08 Thread Rad Gruchalski
Greg, All you need to do is tell Spark that the master is mesos://…, as in the example from June. It’s all nicely documented here: http://spark.apache.org/docs/latest/running-on-mesos.html I’d suggest running in coarse mode as fine grained is a bit choppy. Best regards,
 Radek

Re: Set LIBPROCESS_IP for frameworks launched with marathon

2016-04-03 Thread Rad Gruchalski
Hi Gmail, AFAIK not. The only way to do so is setting up the env variable as you do now. Kind regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Re: Launch docker container from Marathon UI

2016-04-26 Thread Rad Gruchalski
Here’s the up to date URL: https://mesosphere.github.io/marathon/docs/generated/api.html#v2_apps_post Best regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/

Re: Launch docker container from Marathon UI

2016-04-26 Thread Rad Gruchalski
It says exactly what the problem is. Start a marathon task with correct environment variables in env and you will be fine. Best regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/

Re: Launch docker container from Marathon UI

2016-04-26 Thread Rad Gruchalski
gt; > > Stefano, > > > > > > The docker run flag --link is intended to connect the container to > > > another running container. I do not know how this would operate in > > > marathon. Perhaps it would be an application group which starts

Re: Custom IPTables rules

2016-04-13 Thread Rad Gruchalski
Hi Alfredo, The only thing you need is: -A FORWARD -o docker0 -j DOCKER -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -i docker0 ! -o docker0 -j ACCEPT -A FORWARD -i docker0 -o docker0 -j ACCEPT Best regards,
 Radek Gruchalski 


Re: Custom IPTables rules

2016-04-13 Thread Rad Gruchalski
the sender immediately. On Wednesday, 13 April 2016 at 22:14, Alfredo Carneiro wrote: > Unfortunately, I am facing some problemseven with my INPUT rules allowing > just some subnetworks, Docker is accepting connections from everywhere. > > On Wed, Apr 13, 2016 at 5:06 PM, Ra

Re: Custom IPTables rules

2016-04-13 Thread Rad Gruchalski
: > Oh man! Really thanks! It worked! > > On Wed, Apr 13, 2016 at 4:57 PM, Rad Gruchalski <ra...@gruchalski.com > (mailto:ra...@gruchalski.com)> wrote: > > Have you tried restarting docker daemon afterwards? > > > > > > > > > > >

Re: Custom IPTables rules

2016-04-13 Thread Rad Gruchalski
CKER (1 references) > target prot opt source destination > > > I hid the INPUT chain because is very big! > > Best Regards, > > On Wed, Apr 13, 2016 at 4:29 PM, Rad Gruchalski <ra...@gruchalski.com > (mailto:ra...@gruchalski.com)> wrote: > &g