Re: Fwd: Unable to run spark examples on mesos 1.0

[max square]

Thanks Stephen. That did the job for me. After adding JAVA_HOME in
hadoop-layout.sh, I was able to run the spark-job successfully.
@mgummelt - I did not set the executor_environment_variables. However, I am
now able to see JAVA_HOME when I print out the 'env' in the driver.


On Fri, Aug 5, 2016 at 1:53 PM, mgumm...@mesosphere.io <
mgumm...@mesosphere.io> wrote:

> What is your --executor-environment-variables set to?
> http://mesos.apache.org/documentation/latest/configuration/
>
> Can you print out your `env` in the driver to verify it has the expected
> JAVA_HOME
>
> On 2016-08-04 12:28 (-0700), max square  wrote:
> > Hey guys ,
> > I was trying out spark 2.0 examples to run on mesos+hadoop cluster but it
> > keep failing with the following error message:-
> >
> > I0803 19:46:53.848696 12494 fetcher.cpp:498] Fetcher Info:
> > > {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/
> 587226cc-bece-422a-bb93-e3ef49075642-S1\/root","items"
> :[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"
> hdfs:\/\/testcluster\/spark-examples_2.11-2.0.0.jar"}},{"
> action":"BYPASS_CACHE","uri":{"extract":true,"value":"hdfs:\
> /\/testcluster\/spark-2.0.0-bin-hdfs-2.6.0-cdh5.7.1.tgz"}}
> ],"sandbox_directory":"\/vol\/mesos\/data\/slaves\/587226cc-
> bece-422a-bb93-e3ef49075642-S1\/frameworks\/587226cc-bece-
> 422a-bb93-e3ef49075642-0017\/executors\/driver-20160803194649-0001\/runs\/
> b1e9a92e-f004-4cdc-b936-52b32593d39f","user":"root"}
> >
> > I0803 19:46:53.850719 12494 fetcher.cpp:409] Fetching URI
> > > 'hdfs://testcluster/spark-examples_2.11-2.0.0.jar'
> >
> > I0803 19:46:53.850731 12494 fetcher.cpp:250] Fetching directly into the
> > > sandbox directory
> >
> > I0803 19:46:53.850746 12494 fetcher.cpp:187] Fetching URI
> > > 'hdfs://testcluster/spark-examples_2.11-2.0.0.jar'
> > > E0803 19:46:53.860776 12494 shell.hpp:106] Command
> > > '/usr/lib/hadoop/bin/hadoop version 2>&1' failed; this is the output:
> > > Error: JAVA_HOME is not set and could not be found.
> > > Failed to fetch 'hdfs://testcluster/spark-examples_2.11-2.0.0.jar':
> Failed
> > > to create HDFS client: Failed to execute '/usr/lib/hadoop/bin/hadoop
> > > version 2>&1'; the command was either not found or exited with a
> non-zero
> > > exit status: 1
> > > Failed to synchronize with agent (it's probably exited)
> >
> >
> > To start out, I tried out the hadoop command which was giving the error
> on
> > the agents and was able to replicate the error. So basically, running
> "sudo
> > -u root /usr/lib/hadoop/bin/hadoop version 2>&1" gave me the same
> JAVA_HOME
> > not set error. After I fixed that and restarted the agents, running the
> > spark example still gave me the same error.
> >
> > I ran the same examples on mesos 0.28.2, and it ran fine.
> >
> > Any help regarding this would be appreciated.
> >
> > *Additional Info :-*
> > mesos version - 1.0.0
> > hadoop version - 2.6.0-cdh5.7.2
> > spark version - 2.0.0
> >
> > Command used to run spark example - ./bin/spark-submit --class
> > org.apache.spark.examples.SparkPi --master mesos://:7077
> > --deploy-mode cluster --executor-memory 2G --total-executor-cores 4
> > hdfs://testcluster/spark-examples_2.11-2.0.0.jar 100
> >
>

A port mapping plugin for CNI

[Avinash Sridharan]

Hi All,
 One of the most used networking mode when users run containers using the
`DockerContainerizer` is docker bridge mode networking. In this mode
containers run in an isolator private address space, and services on the
container are exposed to the outside world using DNAT.

As we move towards the unified containerizer becoming the de-facto
container runtime, and users start running their docker containers on the
`MesosContainerizer`, the expectation of exposing services running on
isolator bridges with DNAT starts becoming a must have.

With the introduction of the `network/cni` isolator we can use CNI plugins
to start attaching containers on the `MesosContainerizer` to different
types of IP networks. Corresponding to docker bridge network, CNI has its
own bridge plugin, however unlike docker bridge networking the CNI bridge
plugin does not provide DNAT services to expose containers on a bridge.
None of the core CNI plugins provide a port mapping functionality, and it
is only recently that there seems to be a push for having port mapping
functionality in a CNI plugin.

We are therefore proposing implementing a CNI plugin that can setup port
mapping rules for different CNI plugins for Mesos. This CNI plugin is
generic enough that it can be used in conjunction with any other CNI
plugin, such as the bridge plugin.

The motivation, design and operational aspects of the plugin have been
captured in this document:
https://docs.google.com/document/d/1ZwXZ_utpxmy9vccYiL0q86efgpWpjmmKLQ0S4Mmz9N4/edit?usp=sharing

Would be great if the community can share their feedback on the proposed
port mapping CNI plugin.

Thanks,
-- 
Avinash Sridharan, Mesosphere
+1 (323) 702 5245

Parameters for the Docker Executor

[Hendrik Haddorp]

Hi,

in mesos.proto [1] it states:
// Allowing arbitrary parameters to be passed to docker CLI.
// Note that anything passed to this field is not guaranteed
// to be supported moving forward, as we might move away from
// the docker CLI.
repeated Parameter parameters = 5;

After failing to get my containers started with parameters I checked the
code in docker.cpp [2] and found this:
  foreach (const Parameter& parameter, dockerInfo.parameters()) {
argv.push_back("--" + parameter.key() + "=" + parameter.value());
  }

So I can actually only use parameters in the form --KEY=VALUE, which
seems to be quite a restriction when looking at all the options the
docker cli has. What is the reason for this limitation and is there any
way around this?

regards,
Hendrik

[1] https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto
[2] https://github.com/apache/mesos/blob/master/src/docker/docker.cpp

Re: Fwd: Unable to run spark examples on mesos 1.0

[mgumm...@mesosphere.io]

What is your --executor-environment-variables set to? 
http://mesos.apache.org/documentation/latest/configuration/

Can you print out your `env` in the driver to verify it has the expected 
JAVA_HOME

On 2016-08-04 12:28 (-0700), max square  wrote: 
> Hey guys ,
> I was trying out spark 2.0 examples to run on mesos+hadoop cluster but it
> keep failing with the following error message:-
> 
> I0803 19:46:53.848696 12494 fetcher.cpp:498] Fetcher Info:
> > {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/587226cc-bece-422a-bb93-e3ef49075642-S1\/root","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"hdfs:\/\/testcluster\/spark-examples_2.11-2.0.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"hdfs:\/\/testcluster\/spark-2.0.0-bin-hdfs-2.6.0-cdh5.7.1.tgz"}}],"sandbox_directory":"\/vol\/mesos\/data\/slaves\/587226cc-bece-422a-bb93-e3ef49075642-S1\/frameworks\/587226cc-bece-422a-bb93-e3ef49075642-0017\/executors\/driver-20160803194649-0001\/runs\/b1e9a92e-f004-4cdc-b936-52b32593d39f","user":"root"}
> 
> I0803 19:46:53.850719 12494 fetcher.cpp:409] Fetching URI
> > 'hdfs://testcluster/spark-examples_2.11-2.0.0.jar'
> 
> I0803 19:46:53.850731 12494 fetcher.cpp:250] Fetching directly into the
> > sandbox directory
> 
> I0803 19:46:53.850746 12494 fetcher.cpp:187] Fetching URI
> > 'hdfs://testcluster/spark-examples_2.11-2.0.0.jar'
> > E0803 19:46:53.860776 12494 shell.hpp:106] Command
> > '/usr/lib/hadoop/bin/hadoop version 2>&1' failed; this is the output:
> > Error: JAVA_HOME is not set and could not be found.
> > Failed to fetch 'hdfs://testcluster/spark-examples_2.11-2.0.0.jar': Failed
> > to create HDFS client: Failed to execute '/usr/lib/hadoop/bin/hadoop
> > version 2>&1'; the command was either not found or exited with a non-zero
> > exit status: 1
> > Failed to synchronize with agent (it's probably exited)
> 
> 
> To start out, I tried out the hadoop command which was giving the error on
> the agents and was able to replicate the error. So basically, running "sudo
> -u root /usr/lib/hadoop/bin/hadoop version 2>&1" gave me the same JAVA_HOME
> not set error. After I fixed that and restarted the agents, running the
> spark example still gave me the same error.
> 
> I ran the same examples on mesos 0.28.2, and it ran fine.
> 
> Any help regarding this would be appreciated.
> 
> *Additional Info :-*
> mesos version - 1.0.0
> hadoop version - 2.6.0-cdh5.7.2
> spark version - 2.0.0
> 
> Command used to run spark example - ./bin/spark-submit --class
> org.apache.spark.examples.SparkPi --master mesos://:7077
> --deploy-mode cluster --executor-memory 2G --total-executor-cores 4
> hdfs://testcluster/spark-examples_2.11-2.0.0.jar 100
> 

Re: Framework for runtime configuration of a mesos provided resource

[DiGiorgio, Mr. Rinaldo S.]

On Aug 4, 2016, at 11:56 PM, haosdent 
> wrote:

Do you start all your Mesos Agents with root user?

We use systemctl on linux, smf on solaris, a plist launchd on OS X ( which has 
severe latency issues)  and just now looking at windows.  When we start the 
task we switch to a non root user.


On Thu, Aug 4, 2016 at 7:54 PM, DiGiorgio, Mr. Rinaldo S. 
> wrote:
Hi,

We have software that runs unmodified on many different operating systems ( 
Linux, Solaris, Windows and OS X). We have found several cases where the mesos 
provided compute resource requires some additional setup since the created 
process is not quite the same as a process created by login. We would prefer to 
have some code apply changes once the resource is allocated and before we start 
our task on the resource. Is there any framework support for this that you 
could recommend or suggest I look at. We don’t want to have out application 
need to be  aware that it is running in a mesos created process.

Some examples:

On OS X several keychain operations fail because the keychain is not created. A 
reader of this list provided a solution that I would like to apply to the node 
before starting the task that requires valid keychains.  You can’t configure 
some of these things before the process is created so using puppet or ansible 
would not help. It needs to be done at runtime.

On Linux and OS X /usr/bin/expect fails due to some issues that may be related 
to environment settings like HOME, LOGNAME … The mesos plugin for jenkins 
provides support for setting of some values before the node is executed.



Rinaldo



--
Best Regards,
Haosdent Huang

Re: Fwd: Unable to run spark examples on mesos 1.0

[Stephen Gran]

Hi,

You'll need to get a working hadoop install before that works.  Try 
adding JAVA_HOME and so forth to hadoop/libexec/hadoop-layout.sh

Cheers,

On 04/08/16 20:28, max square wrote:
> Hey guys ,
> I was trying out spark 2.0 examples to run on mesos+hadoop cluster but
> it keep failing with the following error message:-
>
> I0803 19:46:53.848696 12494 fetcher.cpp:498] Fetcher Info:
> 
> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/587226cc-bece-422a-bb93-e3ef49075642-S1\/root","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"hdfs:\/\/testcluster\/spark-examples_2.11-2.0.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"hdfs:\/\/testcluster\/spark-2.0.0-bin-hdfs-2.6.0-cdh5.7.1.tgz"}}],"sandbox_directory":"\/vol\/mesos\/data\/slaves\/587226cc-bece-422a-bb93-e3ef49075642-S1\/frameworks\/587226cc-bece-422a-bb93-e3ef49075642-0017\/executors\/driver-20160803194649-0001\/runs\/b1e9a92e-f004-4cdc-b936-52b32593d39f","user":"root"}
>
> I0803 19:46:53.850719 12494 fetcher.cpp:409] Fetching URI
> 'hdfs://testcluster/spark-examples_2.11-2.0.0.jar'
>
> I0803 19:46:53.850731 12494 fetcher.cpp:250] Fetching directly into
> the sandbox directory
>
> I0803 19:46:53.850746 12494 fetcher.cpp:187] Fetching URI
> 'hdfs://testcluster/spark-examples_2.11-2.0.0.jar'
> E0803 19:46:53.860776 12494 shell.hpp:106] Command
> '/usr/lib/hadoop/bin/hadoop version 2>&1' failed; this is the output:
> Error: JAVA_HOME is not set and could not be found.
> Failed to fetch 'hdfs://testcluster/spark-examples_2.11-2.0.0.jar':
> Failed to create HDFS client: Failed to execute
> '/usr/lib/hadoop/bin/hadoop version 2>&1'; the command was either
> not found or exited with a non-zero exit status: 1
> Failed to synchronize with agent (it's probably exited)
>
>
> To start out, I tried out the hadoop command which was giving the error
> on the agents and was able to replicate the error. So basically, running
> "sudo -u root /usr/lib/hadoop/bin/hadoop version 2>&1" gave me the same
> JAVA_HOME not set error. After I fixed that and restarted the agents,
> running the spark example still gave me the same error.
>
> I ran the same examples on mesos 0.28.2, and it ran fine.
>
> Any help regarding this would be appreciated.
>
> *Additional Info :-*
> mesos version - 1.0.0
> hadoop version - 2.6.0-cdh5.7.2
> spark version - 2.0.0
>
> Command used to run spark example - ./bin/spark-submit --class
> org.apache.spark.examples.SparkPi --master mesos://:7077
> --deploy-mode cluster --executor-memory 2G --total-executor-cores 4
> hdfs://testcluster/spark-examples_2.11-2.0.0.jar 100
>
>
>
>
>
>

-- 
Stephen Gran
Senior Technical Architect

picture the possibilities | piksel.com