Re: Directory mounted in job not visible on host

[Tobias Pfeiffer]

Jie,

that's great, thank you! Looking forward to that functionality!

Thanks
Tobias

On Tue, Oct 24, 2017 at 12:20 AM, Jie Yu <yujie@gmail.com> wrote:

> Tobias,
>
> By default, Mesos marks all mounts in a Mesos container as slave mount.
> Therefore, the mount propagation is from host to container, but not
> container to host.
>
> I am actually working on a patch chain to enable bidirectional mount
> propagation:
> https://issues.apache.org/jira/browse/MESOS-7306
>
> In particular, see the proposed API in this patch:
> https://reviews.apache.org/r/63213/
>
> That'll help achieve your goal. Stay tuned and we'll land this in next
> Mesos release.
>
> - Jie
>
> On Mon, Oct 23, 2017 at 1:00 AM, Tobias Pfeiffer <t...@preferred.jp> wrote:
>
>> Hi,
>>
>> in my Mesos job (Mesos containerizer) I am mounting a squashfs image file
>> to some directory on the file system and can access the directory and its
>> contents fine from within that job. However, on the Mesos host (i.e., not
>> in the job itself) that directory does not appear in the output of the
>> `mount` command and when inspecting the directory, it is empty. In
>> particular, my Mesos job launches a Docker container and mounts that
>> previously mounted directory as a volume (don't ask ...), but in the Docker
>> container that volume is also empty.
>>
>> I am wondering if there is any way that I could make a mount operation
>> performed by a job visible to the outside world?
>>
>> Thanks
>> Tobias
>>
>>
>


-- 
Tobias Pfeiffer, Lead Engineer
Preferred Networks Inc, <https://www.preferred-networks.jp/>

Directory mounted in job not visible on host

[Tobias Pfeiffer]

Hi,

in my Mesos job (Mesos containerizer) I am mounting a squashfs image file
to some directory on the file system and can access the directory and its
contents fine from within that job. However, on the Mesos host (i.e., not
in the job itself) that directory does not appear in the output of the
`mount` command and when inspecting the directory, it is empty. In
particular, my Mesos job launches a Docker container and mounts that
previously mounted directory as a volume (don't ask ...), but in the Docker
container that volume is also empty.

I am wondering if there is any way that I could make a mount operation
performed by a job visible to the outside world?

Thanks
Tobias

Data locality in Mesos

[Tobias Pfeiffer]

Hi,

one of the major selling points of HDFS is (was?) that it is possible to
schedule a Hadoop job close to where the data that it operates on is.  I am
not using HDFS, but I was wondering if/how Mesos supports an approach to
schedule a job to a machine that has a certain file/dataset already locally
as opposed to scheduling it to a machine that would have to access it via
the network or download to the local disk first.

I was wondering if Mesos attributes could be used:  I could have an
attribute `datasets` of type `set` and then node A could have {dataset1,
dataset17, dataset3} and node B could have {dataset17, dataset5} and during
scheduling I could decide based on this attribute where to run a task.
However, I was wondering if there are dynamic changes of such attributes
possible.  Imagine that node A deletes dataset17 from the local cache and
downloads dataset5 instead, then I would like to update the `datasets`
attribute dynamically, but without affecting the jobs that are running on
node A.  Is such a thing possible?

Is there an approach other than attributes to describe the data that
resides on a node in order to achieve data locality?

Thanks
Tobias

Re: [Req]Starting Japan User Group

[Tobias Pfeiffer]

Hi,

On Wed, May 10, 2017 at 2:06 AM, Kitayama, Shingo 
wrote:
>
> Thank you for your approval to create MUG in Tokyo.
>
> At once we are just starting to extend values and make collaboration thr
> Mesos in Japan.
>
> At the begging after making the first meetup clear, we will announce the
> detail to you.
>

Apparently yesterday was the first meetup <
https://mesos.connpass.com/event/57331/>?
It's a pity you did not announce the Conpass link etc. on the list.

Kind regards,
Tobias

Re: [Req]Starting Japan User Group

[Tobias Pfeiffer]

Hi Shingo,

On Tue, May 9, 2017 at 4:50 PM, Kitayama, Shingo <shingo.kitay...@hpe.com>
wrote:
>
> Additionally I and my team members are planning to held the Mesos Meetup
> in Tokyo and to extend its value continuously.
>
> In this meetup, we will foster the adoption of Mesos, exchange tips or
> issues, and make up collaborative solutions with participants.
>

Great news, happy to hear that!  I am very much interested in that, please
let us know where the meetups will take place, how to register, etc. –
thank you!

Kind regards,
Tobias

-- 
Tobias Pfeiffer, Lead Engineer
Preferred Networks Inc, <https://www.preferred-networks.jp/>

Re: Error running Application in Marathon

[Tobias Pfeiffer]

Hi,

On Wed, Jan 11, 2017 at 7:20 PM, Joaquin Alzola 
wrote:
>
> #export MESOS_NATIVE_JAVA_LIBRARY=/usr/loca/lib/libmesos-1.1.0.so
>

You have a typo here, local is missing an l.

Tobias

Re: Proposal: mesosadm, the command to bootstrap the mesos cluster.

[Tobias Pfeiffer]

Hi,

On Thu, Dec 15, 2016 at 9:31 AM, Alex Rukletsov  wrote:

> I have a different opinion on this. Several years ago I came across the
> concept of "mean wizards" — any helpers that hide away important steps from
> the user and hence do not give them opportunity to learn how things
> actually work.
>

On a related note: I think most production setups will use some kind of
configuration management tool, Ansible, cfengine etc., often with an
declarative instead of imperative style. If you have a "command to set
something up", if calling that command is not an idempotent operation then
handling it in a declarative script becomes more difficult. You end up
writing "if some file (generated as a side effect of that program) is not
present, then run command, otherwise do nothing" instead of "ensure that
some configuration file is in place with a well-defined content".

While I think any tool that simplifies "getting started" is great, I think
this may turn all too quickly into a "mean wizard" thing that does some
black magic behind the scenes and that it becomes basically impossible to
setup Mesos *without* the wizard.

Tobias

Re: Mesos making offers for no CPU

[Tobias Pfeiffer]

Hi Christopher,

do you mean, you *never* receive updates about other nodes or you just
don't receive them regularly? If you receive an offer initially and you
don't accept or decline it, you will not receive that offer again. You can
cache it for some time locally and then accept it at a later point in time.
Or you decline it and then you will receive the offer again some seconds
later. In principle.

Hope that helps,
Tobias


On Mon, Nov 28, 2016 at 12:31 PM, Christopher Hunt <
christopher.h...@lightbend.com> wrote:

> Hi there,
>
> I’ve been using Mesos for a few months now and I’m a little mystified by
> Mesos sending my framework resource offers with no CPU in them.
>
> I cannot see much utility of Mesos sending such resource offers, but that
> aside, if my framework declines them then why doesn’t Mesos send resource
> offers for other nodes where there are resources? As it stands, Mesos
> appears to keep sending my framework the same resource offer. This
> ultimately fails my framework given that it cannot proceed with launching a
> task.
>
> I can see *why* the resource offer has no CPU - there isn’t any available.
> I’m curious to understand why Mesos cannot move on and send resources for
> other nodes.
>
> I’m using DC/OS 1.8 which I believe uses Mesos 1.0.
>
> Thanks for any info.
>
> Kind regards,
> Christopher
>
> Christopher Hunt
> *Technical Lead, Lightbend Production Suite*
> @huntchr
> UTC+10
>
>

Re: Path vs Mount disk resources

[Tobias Pfeiffer]

Hi Jie,

On Mon, Nov 21, 2016 at 4:45 PM, Jie Yu  wrote:

> The documentation states "non performance-critical applications" and
>> "should only be done in a testing or staging environment" about Path type
>> resources. Is there any particular reason for that?
>
>
> I think the documentation is not very precise. Path disk resource is the
> default disk resource, and has been used in prod for years for task's logs
> and other stuffs in its sandbox.
>

OK, great, thank you very much!

Tobias

Mesos containerizer & isolation

[Tobias Pfeiffer]

Hi,

I asked this question also yesterday in the #mesos channel on IRC, but I
guess due to timezone differences there were not many people awake and/or
working, sorry for reposting. (Maybe someone answered after I left, but it
seems that the IRC bot is only archiving channel joins/leaves? ->
http://wilderness.apache.org/channels/?f=apache-syncope/2016-11-02)

My question is about the Mesos containerizer. I want to run code using the
Mesos GPU support and the docs state that this is currently only supported
by the Mesos containerizer. So my understanding of using the Mesos
containerizer with Docker images is that
- the content of the Docker images is unpacked to the filesystem (using one
of the provisioner backends, such as "copy" or "overlay")
- the user's command is executed in a chroot in that directory.
Is that correct?

The first thing I noticed is (besides a much higher latency due to the
image provisioning process) that `ps aux` and `hostname` expose details of
the host system, so I was wondering about the level of isolation that I can
achieve with the Mesos containerizer, as opposed to running in a Docker
container. In particular:
- Is it possible to hide host processes from the container?
- Is it possible to run processes that open network ports (possibly already
open on the host system) and have them mapped to different ports on the
host system, just as with Docker's `-p`?
- I have a USER directive in my Dockerfile in order for the CMD to be
executed as that user, but that does not seem to be supported (yet?) by the
Docker image provider. Is there any method (except `sudo`/`setuser`) to
achieve running as a user present in the image's /etc/fstab?
- I may have to run untrusted code, so can I make sure that users cannot
break out of the chroot? What about UID namespacing, so that root in the
chroot does not become root on the host system when breaking out?

Thanks for your help
Tobias

Re: Identify submitted TaskInfo and TaskStatus?

[Tobias Pfeiffer]

Hi Hendrik,

On Wed, Aug 31, 2016 at 6:05 PM, Hendrik Haddorp 
wrote:
>
> I'm setting the TaskId for the task that I start and then see the same
> TaskId in the update. So that is quite easy to match.
>

I see, I guess that is the straightforward and trivial way to do it ;-)
Thank you!

Tobias

Re: Low-latency task submission

[Tobias Pfeiffer]

Hi Alexander,

thanks for your suggestion.

On Wed, Aug 31, 2016 at 9:11 PM, Alexander Rojas 
wrote:
>
> There are some things you can try, one would be to change the default
> value of the master’s `allocation_interval` flag, which is 1. This will
> increase the rate at which offers are being generated.
>

I see. Do you think that in general this is the right approach, processing
all tasks that have piled up in the last second within the
`resourceOffers()` method?
Also, I read somewhere that on accepting an offer, all other offers will be
declined; but is this only true for offers from the same slave?

Thanks,
Tobias

Low-latency task submission

[Tobias Pfeiffer]

Hi,

as written in a previous thread, I am trying to build a system where users
submit code and this code is run in a Docker container. I want to keep
latency low, so the time between "user submits request" and "task is
submitted" should be short.
First I tried to (a) buffer arriving tasks in an internal queue and
whenever `resourceOffers()` was called, process that queue. All not-used
offers were declined. That worked, but the time between resource offerings
was rather long, so that I tried to find a different way.
The next try (b) was to instead buffer the offers I received and process
new requests immediately from that buffer. However, the drawbacks were that
1) no other frameworks received those resource offers while they are in my
buffer, and 2) when other frameworks finished a task, a lot of resource
offers for the same slave piled up (like "0.5 CPUs and 500 MB memory", then
"0.8 CPUs and 0 MB memory" etc.).
I guess once I have understood these mechanisms I can work around them in
my scheduler and do *something* (like "buffer two resource offers and
decline the rest" or so), but is there any best practice approach for that?

Thanks,
Tobias