How to get the Mesos Spark Framework to use multiple disks

[Jeff Kubina]

>From the spark documentation

on spark.local.dir or SPARK_LOCAL_DIRS options it looks like the Mesos
Spark Framework can be configured to use multiple disks.

spark.local.dir Directory to use for "scratch" space in Spark, including
map output files and RDDs that get stored on disk. This should be on a
fast, local disk in your system. *It can also be a comma-separated list of
multiple directories on different disks.* NOTE: *In Spark 1.0 and later
this will be overridden by SPARK_LOCAL_DIRS (Standalone, Mesos)* or
LOCAL_DIRS (YARN) environment variables set by the cluster manager.

Does anyone have experience doing this or have guidance on how to configure
this?  Would I need to configure multiple disks for persistent volumes for
the Spark Framework to use the disks?

Re: Is it possible to configure a mesos agent to use multiple work directories?

[Jeff Kubina]

Thanks, that is what I thought.

Why: To spread the I/O-workload of some frameworks across many disks.

-- 
Jeff Kubina
410-988-4436


On Wed, Nov 22, 2017 at 2:21 PM, Vinod Kone <vinodk...@apache.org> wrote:

> No. Why do you need that?
>
> On Wed, Nov 22, 2017 at 10:42 AM, Jeff Kubina <jeff.kub...@gmail.com>
> wrote:
>
>> Is it possible to configure a mesos agent to use multiple work
>> directories (the work_dir parameter)?
>>
>>
>

Is it possible to configure a mesos agent to use multiple work directories?

[Jeff Kubina]

Is it possible to configure a mesos agent to use multiple work directories
(the work_dir parameter)?

Re: Format for attributes with no value

[Jeff Kubina]

https://issues.apache.org/jira/browse/MESOS-7824

-- 
Jeff Kubina
410-988-4436


On Fri, Jul 14, 2017 at 9:03 AM, James Peach <jor...@gmail.com> wrote:

>
> > On Jul 13, 2017, at 1:41 PM, Jeff Kubina <jeff.kub...@gmail.com> wrote:
> >
> > I want to know the format for an empty attribute in the list format for
> the mesos-slave --attributes option. If I have an attribute, say key2, with
> no value would it be "mesos-slave --attributes
> key1:value1;key2;key3:value3" or "mesos-slave --attributes
> key1:value1;key2:;key3:value3 or does it not matter?
>
> As I said before, I don't think there is a way to have an empty attribute
> value.
>
> $ sudo /opt/mesos/agent "--attributes=key1:value1;key2:;key3:value3"
> ...
> F0714 06:16:49.332021 16167 attributes.cpp:145] Invalid attribute
> key:value pair 'key2:'
>
> $ sudo /opt/mesos/agent "--attributes=key1:value1;key2;key3:value3"
> ...
> F0714 06:17:58.916723 16297 attributes.cpp:145] Invalid attribute
> key:value pair 'key2'
>
> Please file a bug at https://issues.apache.org/jira/projects/MESOS
>
> >
> > --
> > Jeff Kubina
> > 410-988-4436
> >
> >
> > On Tue, Jul 11, 2017 at 5:20 AM, Oeg Bizz <oegb...@yahoo.com> wrote:
> > James,
> >   If you need an empty attribute as default for mesos, just create an
> empty file with the '?' in front of it and save it in the
> /etc/mesos- directory.  For instance, if you want to
> enable authentication and want to pass the --authenticate attribute then
> create an empty file called
> > /etc/mesos-master/?authenticate.
> >
> > Not sure if that is what you meant with your question,
> >
> > Oscar
> >
> >
> > On Tuesday, July 11, 2017, 12:53:37 AM EDT, James Peach <
> jor...@gmail.com> wrote:
> >
> >
> >
> > > On Jul 7, 2017, at 4:46 PM, Jeff Kubina <jeff.kub...@gmail.com> wrote:
> > >
> > > When setting an attribute with no value of a mesos-agent is the colon
> needed, optional, or must it be omitted? It's not clear from the
> documentation. For example, which line or lines below are correct?
> > >
> > > att1:val1;att2;att3:val3
> > >
> > > att1:val1;att2:;att3:val3
> >
> >
> > I don't see a way to express an empty attribute at all :(
> >
>
>

Re: Format for attributes with no value

[Jeff Kubina]

I want to know the format for an empty attribute in the list format for the
mesos-slave --attributes option. If I have an attribute, say key2, with no
value would it be "mesos-slave --attributes key1:value1;key2;key3:value3"
or "mesos-slave --attributes key1:value1;key2:;key3:value3 or does it not
matter?

-- 
Jeff Kubina
410-988-4436


On Tue, Jul 11, 2017 at 5:20 AM, Oeg Bizz <oegb...@yahoo.com> wrote:

> James,
>   If you need an empty attribute as default for mesos, just create an
> empty file with the '?' in front of it and save it in the
> /etc/mesos- directory.  For instance, if you want to
> enable authentication and want to pass the --authenticate attribute then
> create an empty file called
> /etc/mesos-master/?authenticate.
>
> Not sure if that is what you meant with your question,
>
> Oscar
>
>
> On Tuesday, July 11, 2017, 12:53:37 AM EDT, James Peach <jor...@gmail.com>
> wrote:
>
>
>
> > On Jul 7, 2017, at 4:46 PM, Jeff Kubina <jeff.kub...@gmail.com> wrote:
> >
> > When setting an attribute with no value of a mesos-agent is the colon
> needed, optional, or must it be omitted? It's not clear from the
> documentation. For example, which line or lines below are correct?
> >
> > att1:val1;att2;att3:val3
> >
> > att1:val1;att2:;att3:val3
>
>
> I don't see a way to express an empty attribute at all :(
>

Is there a way to get a dump of all the configuration settings of a master or agent?

[Jeff Kubina]

Is there a way to get a dump of all the configuration settings of a master
or agent?

Format for attributes with no value

[Jeff Kubina]

When setting an attribute with no value of a mesos-agent is the colon
needed, optional, or must it be omitted? It's not clear from the
documentation. For example, which line or lines below are correct?

att1:val1;att2;att3:val3

att1:val1;att2:;att3:val3