Hi Igor,
The best way I know of is with Marathon.
* Placement constraint: you could combine constraints in Marathon. Like:
"constraints": [
["hostname", "UNIQUE"],
["hostname", "LIKE", "host1|host2|host3"]
]
https://groups.google.com/forum/#!topic/marathon-framework/hfLUw3TIw2I
* You would have to use a workaround to deal with a dynamically sized
cluster: set the number of instances to be greater than the expected
cluster size.
https://jira.mesosphere.com/browse/MARATHON-3791?focusedCommentId=79976&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-79976
As the commenter notes, it's not ideal, it's just a workaround.
Susan
On Sat, Jan 20, 2018 at 8:33 AM, igor.berman <igor.ber...@gmail.com> wrote:
> Hi,
> wanted to get some advice regarding managing external shuffle service in
> mesos environments
>
> In spark documentation the Marathon is mentioned, however there is very
> limited documentation.
> I've tried to search for some documentation and it's seems not too
> difficult
> to configure it under Marathon(e.g.
> https://github.com/NBCUAS/dcos-spark-shuffle-service/
> blob/master/marathon/mesos-shuffle-service.json),
> however I see few problems:
>
> There is no clear way to deploy some application in mesos on every node
> see https://jira.mesosphere.com/browse/MARATHON-3791
> * it's not possible to guarantee on which nodes shuffle service application
> will be placed(it's possible to guarantee with mesos unique constrain that
> only 1 shuffle service instance will be placed on some node)
> * cluster that has dynamic nodes joining/leaving - the config of shuffle
> service must be adjusted(specifically number of instances config)
>
> So any production ops advices will be welcome
> Igor
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>
--
Susan X. Huynh
Software engineer, Data Agility
xhu...@mesosphere.com
