Re: mesos, big data and service discovery

[Shuai Lin]

What about specifying all non-local instances as "backup" in haproxy.cfg?
This way haproxy would only direct traffic to the local instance as long as
the local instance is alive.

For example, if you plan to use the haproxy-marathon-bridge script, you can
modify this line to achieve that:
https://github.com/mesosphere/marathon/blob/8b3ce8844dcc53055345914ef11019789dd843cf/bin/haproxy-marathon-bridge#L162
.


On Thu, Dec 31, 2015 at 1:56 AM, vincent gromakowski <
vincent.gromakow...@gmail.com> wrote:

> I am currently using mesos as a big data backend for spark, cassandra,
> kafka and elasticsearch but I cannot find a good overall design regarding
> service discovery. I explain:
> Generally, the service discovery is managed by a HAproxy instance on each
> node which redirect trafic from service ports to real assigned network
> ports. Currently I am not using it because the cluster is quite small and I
> don't need to deploy lots of service but I am thinking on futur design that
> will allows me to scale.
> The problem with HAproxy dealing with all network trafic is that I am
> afraid it will break the data locality which is so important in the big
> data world regarding performances.
> For example when Spark tries to connect to elasticsearch, it will discover
> the elasticsearch topology and try to launch tasks next to elasticsearch
> shards. If HAproxy intercept network flows, what would be the result ?
> Will HAproxy masquarade the elasticsearch  IP/ports ? Same thing for Kafka
> and Cassandra ?
>
> I assume it depends on each connector but it's very hard to find any
> information. Thanks for your help if you have any experience in it.
> Regards
>
>
>

Re: The issue of "Failed to shutdown socket with fd xx: Transport endpoint is not connected" on Mesos master

[Nan Xiao]

Hi Avinash,

Sorry for my unclear expression!

The root cause is not related to k8s, but the CentOS which k8s is running on.
It is related to iptables. After executing "iptables -F", it works!

Best Regards
Nan Xiao


On Wed, Dec 30, 2015 at 11:41 PM, Avinash Sridharan
 wrote:
> Thanks for the update Nan. k8s enabling firewall rules that would block
> traffic to the master seems a bit odd. Looks like a bug to me, in the head
> of the branch. If you are able to reproduce it consistently, could you file
> an issue against kubernetes mesos.
>
> regards,
> Avinash
>
> On Tue, Dec 29, 2015 at 11:04 PM, Nan Xiao  wrote:
>>
>> Hi Avinash,
>>
>> Thanks very much for your reply!
>>
>> The root cause has been found: the k8s server has enabled the iptables
>> which blocks connection from
>> Mesos master; after disable it, it works!
>>
>> Best Regards
>> Nan Xiao
>>
>>
>> On Wed, Dec 30, 2015 at 3:22 AM, Avinash Sridharan
>>  wrote:
>> > lsof command will show only actively opened file descriptors. So if you
>> > ran
>> > the command after seeing the error logs in the master, most probably the
>> > master had already closed this fd. Just throwing a few other things to
>> > look
>> > at, that might give some more insights.
>> >
>> > * Run the "netstat -na" and netstat -nt" commands on the master and the
>> > kubernetes master node to make sure that the master is listening to the
>> > right port, and the k8s scheduler is trying to connect to the right
>> > port.
>> > From the logs it does look like the master is receiving the registration
>> > request, so there shouldn't be a network configuration issue here.
>> > * Make sure there are no firewall rules getting turned on in your
>> > cluster
>> > since it looks like the k8s scheduler is not able to connect to the
>> > master
>> > (though it was able to register the first time).
>> >
>> > On Tue, Dec 29, 2015 at 1:37 AM, Nan Xiao 
>> > wrote:
>> >>
>> >> BTW, using "lsof" command finds there are only 16 file descriptors. I
>> >> don't know why Mesos
>> >> master try to close "fd 17".
>> >> Best Regards
>> >> Nan Xiao
>> >>
>> >>
>> >> On Tue, Dec 29, 2015 at 11:32 AM, Nan Xiao 
>> >> wrote:
>> >> > Hi Klaus,
>> >> >
>> >> > Firstly, thanks very much for your answer!
>> >> >
>> >> > The km processes are all live:
>> >> > root 129474 128024  2 22:26 pts/000:00:00 km apiserver
>> >> > --address=15.242.100.60 --etcd-servers=http://15.242.100.60:4001
>> >> > --service-cluster-ip-range=10.10.10.0/24 --port=
>> >> > --cloud-provider=mesos --cloud-config=mesos-cloud.conf
>> >> > --secure-port=0
>> >> > --v=1
>> >> > root 129509 128024  2 22:26 pts/000:00:00 km
>> >> > controller-manager --master=15.242.100.60: --cloud-provider=mesos
>> >> > --cloud-config=./mesos-cloud.conf --v=1
>> >> > root 129538 128024  0 22:26 pts/000:00:00 km scheduler
>> >> > --address=15.242.100.60 --mesos-master=15.242.100.56:5050
>> >> > --etcd-servers=http://15.242.100.60:4001 --mesos-user=root
>> >> > --api-servers=15.242.100.60: --cluster-dns=10.10.10.10
>> >> > --cluster-domain=cluster.local --v=2
>> >> >
>> >> > All the logs are also seem OK, except the logs from scheduler.log:
>> >> > ..
>> >> > I1228 22:26:37.883092  129538 messenger.go:381] Receiving message
>> >> > mesos.internal.InternalMasterChangeDetected from
>> >> > scheduler(1)@15.242.100.60:33077
>> >> > I1228 22:26:37.883225  129538 scheduler.go:374] New master
>> >> > master@15.242.100.56:5050 detected
>> >> > I1228 22:26:37.883268  129538 scheduler.go:435] No credentials were
>> >> > provided. Attempting to register scheduler without authentication.
>> >> > I1228 22:26:37.883356  129538 scheduler.go:928] Registering with
>> >> > master: master@15.242.100.56:5050
>> >> > I1228 22:26:37.883460  129538 messenger.go:187] Sending message
>> >> > mesos.internal.RegisterFrameworkMessage to master@15.242.100.56:5050
>> >> > I1228 22:26:37.883504  129538 scheduler.go:881] will retry
>> >> > registration in 1.209320575s if necessary
>> >> > I1228 22:26:37.883758  129538 http_transporter.go:193] Sending
>> >> > message
>> >> > to master@15.242.100.56:5050 via http
>> >> > I1228 22:26:37.883873  129538 http_transporter.go:587] libproc target
>> >> > URL
>> >> >
>> >> > http://15.242.100.56:5050/master/mesos.internal.RegisterFrameworkMessage
>> >> > I1228 22:26:39.093560  129538 scheduler.go:928] Registering with
>> >> > master: master@15.242.100.56:5050
>> >> > I1228 22:26:39.093659  129538 messenger.go:187] Sending message
>> >> > mesos.internal.RegisterFrameworkMessage to master@15.242.100.56:5050
>> >> > I1228 22:26:39.093702  129538 scheduler.go:881] will retry
>> >> > registration in 3.762036352s if necessary
>> >> > I1228 22:26:39.093765  129538 http_transporter.go:193] Sending
>> >> > message
>> >> > to master@15.242.100.56:5050 via http
>> >> > I1228 22:26:39.093847  129538 http_transporter.go:587] libproc target
>> >> > URL
>> >> >
>> >> > http://15.242.100.56:5050/master/mesos.internal.RegisterFr

Re: More filters on /master/tasks enpoint and filters on /master/state

[Adam Bordelon]

See also:
https://issues.apache.org/jira/browse/MESOS-2258 - Enable filtering of task
information in master/state.json
https://issues.apache.org/jira/browse/MESOS-2353 - Improve performance of
the state.json endpoint for large clusters.
https://issues.apache.org/jira/browse/MESOS-2157 - Add /master/slaves and
/master/frameworks/{framework}/tasks/{task} endpoints
https://docs.google.com/document/d/1dU4e8V6CbomWqHae4FnawZdgK131Iu_YN9uRxZ_y-fw/edit#heading=h.s63wu75ck9qy
- Unraveling of the /state.json endpoint

cc: Alexander, MPark

On Mon, Dec 28, 2015 at 6:20 PM, Klaus Ma  wrote:

> +1
>
> It'll also reduce master's workload; but instead of label, I'd like to
> make master simpler: return tasks page by page and let framework/dashboard
> to filter it themself.
>
>
> 
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> Platform Symphony/DCOS Development & Support, STG, IBM GCG
> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>
> On Tue, Dec 29, 2015 at 6:09 AM, Diogo Gomes  wrote:
>
>> Hi guys, I would like your opinion about a future feature proposal.
>>
>> Currently, we can use HTTP API to list all our tasks running in our
>> cluster using /master/tasks, but you have to list all tasks or limit/offset
>> this list, we cannot filter this. I would like to filter this, using
>> labels, for example. The use case will be to use mesos to fill our load
>> balancer with tasks data.
>>
>>
>> Marathon currently provides something like this, but only for his tasks,
>> using /v2/apps/?label=[key]==[value]
>>
>>
>> Diogo Gomes
>>
>
>

Re: mesos-master v0.26 crashes for quorum 0

[Adam Bordelon]

You should never specify a quorum of 0.
For 1 master, you specify quorum of 1.
For 3 masters, quorum is 2.
For 5 masters, quorum is 3.
For 7 masters, quorum is 4.
The quorum dictates how many masters (log replicas) have to agree on a fact
to win a vote. If you have a quorum of 0, then no masters vote, so nobody
wins. On a related note, you should always have an odd number of masters,
so that the vote is never tied.

I will admit that the master shouldn't crash with --quorum=0; it should
just exit with an error that quorum must be >=1. Want to file a JIRA?

On Tue, Dec 29, 2015 at 3:43 PM, Mehrotra, Ashish 
wrote:

> Hi All,
>
> I am running Centos 7.1, zookeeper version 3.4.7 and Mesos version 0.26.0.
> After starting the zookeeper, when I tried to start to start the
> meson-server with quorum 0 (everything being run on the same machine, not
> as local but distributed set up), the server crashed.
> This happened immediately after the fresh installs.
> When I changed quorum=1, the mesos master ran fine and the slave could get
> connected.
> Then on restarting the mesos master, there was no issue. * The issue was
> seen the very first time only.*
> The error stack is incomprehensible.
>
> Anyone seen this issue previously?
> The error log was —
>
> [root@abc123 build]# ./bin/mesos-master.sh --ip=10.10.10.118
> --work_dir=/var/lib/mesos --zk=zk://10.10.10.118:2181/mesos *--quorum=0*
> I1229 13:41:24.925851  3345 main.cpp:232] Build: 2015-12-29 12:29:36 by
> root
> I1229 13:41:24.925983  3345 main.cpp:234] Version: 0.26.0
> I1229 13:41:24.929131  3345 main.cpp:255] Using 'HierarchicalDRF' allocator
> I1229 13:41:24.953929  3345 leveldb.cpp:176] Opened db in 24.529078ms
> I1229 13:41:24.955523  3345 leveldb.cpp:183] Compacted db in 1.525191ms
> I1229 13:41:24.955688  3345 leveldb.cpp:198] Created db iterator in
> 107413ns
> I1229 13:41:24.955724  3345 leveldb.cpp:204] Seeked to beginning of db in
> 4553ns
> I1229 13:41:24.955737  3345 leveldb.cpp:273] Iterated through 0 keys in
> the db in 224ns
> I1229 13:41:24.956120  3345 replica.cpp:780] Replica recovered with log
> positions 0 -> 0 with 1 holes and 0 unlearned
> I1229 13:41:24.961802  3345 main.cpp:464] Starting Mesos master
> I1229 13:41:24.965438  3345 master.cpp:367] Master
> a38658f7-89c1-4b1f-84f9-5796234b2104 (localhost) started on
> 10.10.10.118:5050
> I1229 13:41:24.965459  3345 master.cpp:369] Flags at startup:
> --allocation_interval="1secs" --allocator="HierarchicalDRF"
> --authenticate="false" --authenticate_slaves="false"
> --authenticators="crammd5" --authorizers="local" --framework_sorter="drf"
> --help="false" --hostname_lookup="true" --initialize_driver_logging="true"
> --ip="10.10.10.118" --log_auto_initialize="true" --logbufsecs="0"
> --logging_level="INFO" --max_slave_ping_timeouts="5" --port="5050"
> --quiet="false" --quorum="0" --recovery_slave_removal_limit="100%"
> --registry="replicated_log" --registry_fetch_timeout="1mins"
> --registry_store_timeout="5secs" --registry_strict="false"
> --root_submissions="true" --slave_ping_timeout="15secs"
> --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false"
> --webui_dir="/home/admin/mesos/build/../src/webui"
> --work_dir="/var/lib/mesos" --zk="zk://10.10.10.118:2181/mesos"
> --zk_session_timeout="10secs"
> I1229 13:41:24.965761  3345 master.cpp:416] Master allowing
> unauthenticated frameworks to register
> I1229 13:41:24.965772  3345 master.cpp:421] Master allowing
> unauthenticated slaves to register
> I1229 13:41:24.965837  3345 master.cpp:458] Using default 'crammd5'
> authenticator
> W1229 13:41:24.965867  3345 authenticator.cpp:513] No credentials
> provided, authentication requests will be refused
> I1229 13:41:24.965881  3345 authenticator.cpp:520] Initializing server SASL
> I1229 13:41:24.966788  3364 log.cpp:238] Attempting to join replica to
> ZooKeeper group
> 2015-12-29 13:41:24,970:3345(0x7f21042f3700):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
> 2015-12-29 13:41:24,970:3345(0x7f21042f3700):ZOO_INFO@log_env@716: Client
> environment:host.name=abc.def.com
> 2015-12-29 13:41:24,970:3345(0x7f21042f3700):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
> 2015-12-29 13:41:24,970:3345(0x7f21042f3700):ZOO_INFO@log_env@724: Client
> environment:os.arch=3.10.0-229.el7.x86_64
> 2015-12-29 13:41:24,970:3345(0x7f21042f3700):ZOO_INFO@log_env@725: Client
> environment:os.version=#1 SMP Fri Mar 6 11:36:42 UTC 2015
> 2015-12-29 13:41:24,970:3345(0x7f21042f3700):ZOO_INFO@log_env@733: Client
> environment:user.name=root
> 2015-12-29 13:41:24,970:3345(0x7f21042f3700):ZOO_INFO@log_env@741: Client
> environment:user.home=/root
> 2015-12-29 13:41:24,970:3345(0x7f21042f3700):ZOO_INFO@log_env@753: Client
> environment:user.dir=/home/admin/mesos/build
> 2015-12-29 13:41:24,970:3345(0x7f21042f3700):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=10.10.10.118:2181 sessionTimeout=1
> watcher=0x7f2110e3f16c sessionI

Re: mesos, big data and service discovery

[John Omernik]

So, no, I don't have Elastic Search in HA Proxy. For each instance of
Elastic Search I have, I specify the ports to user (a range) Now, Spark
can't do service discovery of Elastic serach the way Elastic Search can, so
that could be a challenge. That said, each ES node can be connected to
directly, so perhaps registering each node and using Mesos-DNS with static
ports. Another option is to have your spark app do a little bit of service
discovery of your own. Perhaps specify a range of ports that Elastic Search
COULD be running on and then some nodes it could be running on and have
Spark go guess and check. Since it's your app, you can put what ever logic
you want. I guess, what I am saying is there is nothing built into Spark or
ES to what you want, but between Mesos-DNS, your ability to customize code
in Spark, there should be a clever way to approach that fits your
Environment.



On Wed, Dec 30, 2015 at 12:10 PM, vincent gromakowski <
vincent.gromakow...@gmail.com> wrote:

> Can you confirm what I understand ? Spark will connect to Elasticsearch
> through the service port (means HApoxy) and then will get direct IP/ports
> for the topology?
>
> 2015-12-30 19:06 GMT+01:00 John Omernik :
>
>> I would say that service discovery is only for those services that don't
>> have a built in method for discovery. When I run Elastic Search, I specify
>> the port range I can start elastic search in, and let it run. If the port
>> is taken, it tries a different one (I am using the Elastic Search for Yarn
>> package running on Apache Myriad).  Since I know which nodes and what port
>> ranges to use, I just add that to my Elastic Search config, and thus HA
>> proxy is not intercepting that traffic.  If I have a front end running in
>> Flask that connects to the ES back end, then I would use Mesos-DNS with
>> HAProxy to solve that problem.  In  addition, Spark as a framework does the
>> service discovery, HA Proxy wouldn't be getting inbetween spark nodes, same
>> with Kafka (I haven't played with Cassandra yet).
>>
>> There is some work being done on IP per container which will help this as
>> well, but all in all, I've found that as long I am some what smart about my
>> frameworks, I can manage them (my cluster isn't huge either).   As things
>> grow, I am hoping to grow into IP per container.
>>
>> John
>>
>>
>> On Wed, Dec 30, 2015 at 11:56 AM, vincent gromakowski <
>> vincent.gromakow...@gmail.com> wrote:
>>
>>> I am currently using mesos as a big data backend for spark, cassandra,
>>> kafka and elasticsearch but I cannot find a good overall design regarding
>>> service discovery. I explain:
>>> Generally, the service discovery is managed by a HAproxy instance on
>>> each node which redirect trafic from service ports to real assigned network
>>> ports. Currently I am not using it because the cluster is quite small and I
>>> don't need to deploy lots of service but I am thinking on futur design that
>>> will allows me to scale.
>>> The problem with HAproxy dealing with all network trafic is that I am
>>> afraid it will break the data locality which is so important in the big
>>> data world regarding performances.
>>> For example when Spark tries to connect to elasticsearch, it will
>>> discover the elasticsearch topology and try to launch tasks next to
>>> elasticsearch shards. If HAproxy intercept network flows, what would be the
>>> result ?  Will HAproxy masquarade the elasticsearch  IP/ports ? Same thing
>>> for Kafka and Cassandra ?
>>>
>>> I assume it depends on each connector but it's very hard to find any
>>> information. Thanks for your help if you have any experience in it.
>>> Regards
>>>
>>>
>>>
>>
>

Re: mesos, big data and service discovery

[vincent gromakowski]

Can you confirm what I understand ? Spark will connect to Elasticsearch
through the service port (means HApoxy) and then will get direct IP/ports
for the topology?

2015-12-30 19:06 GMT+01:00 John Omernik :

> I would say that service discovery is only for those services that don't
> have a built in method for discovery. When I run Elastic Search, I specify
> the port range I can start elastic search in, and let it run. If the port
> is taken, it tries a different one (I am using the Elastic Search for Yarn
> package running on Apache Myriad).  Since I know which nodes and what port
> ranges to use, I just add that to my Elastic Search config, and thus HA
> proxy is not intercepting that traffic.  If I have a front end running in
> Flask that connects to the ES back end, then I would use Mesos-DNS with
> HAProxy to solve that problem.  In  addition, Spark as a framework does the
> service discovery, HA Proxy wouldn't be getting inbetween spark nodes, same
> with Kafka (I haven't played with Cassandra yet).
>
> There is some work being done on IP per container which will help this as
> well, but all in all, I've found that as long I am some what smart about my
> frameworks, I can manage them (my cluster isn't huge either).   As things
> grow, I am hoping to grow into IP per container.
>
> John
>
>
> On Wed, Dec 30, 2015 at 11:56 AM, vincent gromakowski <
> vincent.gromakow...@gmail.com> wrote:
>
>> I am currently using mesos as a big data backend for spark, cassandra,
>> kafka and elasticsearch but I cannot find a good overall design regarding
>> service discovery. I explain:
>> Generally, the service discovery is managed by a HAproxy instance on each
>> node which redirect trafic from service ports to real assigned network
>> ports. Currently I am not using it because the cluster is quite small and I
>> don't need to deploy lots of service but I am thinking on futur design that
>> will allows me to scale.
>> The problem with HAproxy dealing with all network trafic is that I am
>> afraid it will break the data locality which is so important in the big
>> data world regarding performances.
>> For example when Spark tries to connect to elasticsearch, it will
>> discover the elasticsearch topology and try to launch tasks next to
>> elasticsearch shards. If HAproxy intercept network flows, what would be the
>> result ?  Will HAproxy masquarade the elasticsearch  IP/ports ? Same thing
>> for Kafka and Cassandra ?
>>
>> I assume it depends on each connector but it's very hard to find any
>> information. Thanks for your help if you have any experience in it.
>> Regards
>>
>>
>>
>

Re: mesos, big data and service discovery

[John Omernik]

I would say that service discovery is only for those services that don't
have a built in method for discovery. When I run Elastic Search, I specify
the port range I can start elastic search in, and let it run. If the port
is taken, it tries a different one (I am using the Elastic Search for Yarn
package running on Apache Myriad).  Since I know which nodes and what port
ranges to use, I just add that to my Elastic Search config, and thus HA
proxy is not intercepting that traffic.  If I have a front end running in
Flask that connects to the ES back end, then I would use Mesos-DNS with
HAProxy to solve that problem.  In  addition, Spark as a framework does the
service discovery, HA Proxy wouldn't be getting inbetween spark nodes, same
with Kafka (I haven't played with Cassandra yet).

There is some work being done on IP per container which will help this as
well, but all in all, I've found that as long I am some what smart about my
frameworks, I can manage them (my cluster isn't huge either).   As things
grow, I am hoping to grow into IP per container.

John


On Wed, Dec 30, 2015 at 11:56 AM, vincent gromakowski <
vincent.gromakow...@gmail.com> wrote:

> I am currently using mesos as a big data backend for spark, cassandra,
> kafka and elasticsearch but I cannot find a good overall design regarding
> service discovery. I explain:
> Generally, the service discovery is managed by a HAproxy instance on each
> node which redirect trafic from service ports to real assigned network
> ports. Currently I am not using it because the cluster is quite small and I
> don't need to deploy lots of service but I am thinking on futur design that
> will allows me to scale.
> The problem with HAproxy dealing with all network trafic is that I am
> afraid it will break the data locality which is so important in the big
> data world regarding performances.
> For example when Spark tries to connect to elasticsearch, it will discover
> the elasticsearch topology and try to launch tasks next to elasticsearch
> shards. If HAproxy intercept network flows, what would be the result ?
> Will HAproxy masquarade the elasticsearch  IP/ports ? Same thing for Kafka
> and Cassandra ?
>
> I assume it depends on each connector but it's very hard to find any
> information. Thanks for your help if you have any experience in it.
> Regards
>
>
>

AW: make slaves not getting tasks anymore

[Mike Michel]

Whitelist seems to be the best option right now. I will try that.

 

thanks

 

Von: Jeremy Olexa [mailto:jol...@spscommerce.com] 
Gesendet: Mittwoch, 30. Dezember 2015 17:22
An: user@mesos.apache.org
Betreff: Re: make slaves not getting tasks anymore

 

Hi Mike,

 

Yes, there is another way besides the maintenance primitives that aren't fully 
complete yet (IMO). If you wish to not schedule anymore jobs, you can remove 
that host from the whitelist on the masters. You might have to engineer this 
for your setup abit, but this is what we do:

 

1) All slaves are discovered and explicitly added to the whitelist

2) On demand (by the operator), a node is REMOVED from the whitelist for some 
time, currently we add the node back after a timeout of 1 hour

3) Wait for jobs to finish on that node, or send SIGUSR1 to mesos-slave process 
to force job termination

 

Of course, there is also satellite, which does all this for you :) 
https://github.com/twosigma/satellite/

 

Hope that helps,
-Jeremy



  _  

From: Mike Michel mailto:mike.mic...@mmbash.de> >
Sent: Wednesday, December 30, 2015 5:43 AM
To: user@mesos.apache.org  
Subject: make slaves not getting tasks anymore 

 

Hi,

 

i need to update slaves from time to time and looking for a way to take them 
out of the cluster but without killing the running tasks. I need to wait until 
all tasks are done and during this time no new tasks should be started on this 
slave. My first idea was to set a constraint „status:online“ for every task i 
start and then change the attribute of the slave to „offline“, restart slave 
process while executer still runs the tasks but it seems if you change the 
attributes of a slave it can not connect to the cluster without rm -rf /tmp 
before which will kill all tasks.

 

Also the maintenance mode seems not to be an option:

 

„When maintenance is triggered by the operator, all agents on the machine are 
told to shutdown. These agents are subsequently removed from the master which 
causes tasks to be updated as TASK_LOST. Any agents from machines in 
maintenance are also prevented from registering with the master.“

 

Is there another way?

 

 

Cheers

 

Mike

mesos, big data and service discovery

[vincent gromakowski]

I am currently using mesos as a big data backend for spark, cassandra,
kafka and elasticsearch but I cannot find a good overall design regarding
service discovery. I explain:
Generally, the service discovery is managed by a HAproxy instance on each
node which redirect trafic from service ports to real assigned network
ports. Currently I am not using it because the cluster is quite small and I
don't need to deploy lots of service but I am thinking on futur design that
will allows me to scale.
The problem with HAproxy dealing with all network trafic is that I am
afraid it will break the data locality which is so important in the big
data world regarding performances.
For example when Spark tries to connect to elasticsearch, it will discover
the elasticsearch topology and try to launch tasks next to elasticsearch
shards. If HAproxy intercept network flows, what would be the result ?
Will HAproxy masquarade the elasticsearch  IP/ports ? Same thing for Kafka
and Cassandra ?

I assume it depends on each connector but it's very hard to find any
information. Thanks for your help if you have any experience in it.
Regards

AW: make slaves not getting tasks anymore

[Mike Michel]

I am using marathon and from shuai lins answer it still seems that maintenance 
mode is not the right option for me. I don’t want to marathon move the tasks to 
another node (phase 1) without user action (restart the task) and it should 
also not just kill the tasks (phase 2).

 

To be concrete: I need to update docker and want to tell users that they need 
to restart their tasks to be moved to a node with the latest docker version. 

 

With MESOS-1739 my „first idea“ would work.

 

Von: Klaus Ma [mailto:klaus1982...@gmail.com] 
Gesendet: Mittwoch, 30. Dezember 2015 13:24
An: user@mesos.apache.org
Betreff: Re: make slaves not getting tasks anymore

 

Hi Mike,

 

Which framework are you using? How about Maintenance's scheduling feature? My 
understanding is that framework show not dispatch task to the maintenance 
agent; so Operator can wait for all tasks finished before taking any action.

 

For "When maintenance is triggered by the operator", it's used when there're 
some tasks took too long time to finish; so Operator can task action to shut 
them down.

 

For the agent restart with new attributes, there's a JIRA (MESOS-1739) about it.






Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer 
Platform Symphony/DCOS Development & Support, STG, IBM GCG 
+86-10-8245 4084 |   klaus1982...@gmail.com |  
 http://k82.me

 

On Wed, Dec 30, 2015 at 7:43 PM, Mike Michel mailto:mike.mic...@mmbash.de> > wrote:

Hi,

 

i need to update slaves from time to time and looking for a way to take them 
out of the cluster but without killing the running tasks. I need to wait until 
all tasks are done and during this time no new tasks should be started on this 
slave. My first idea was to set a constraint „status:online“ for every task i 
start and then change the attribute of the slave to „offline“, restart slave 
process while executer still runs the tasks but it seems if you change the 
attributes of a slave it can not connect to the cluster without rm -rf /tmp 
before which will kill all tasks.

 

Also the maintenance mode seems not to be an option:

 

„When maintenance is triggered by the operator, all agents on the machine are 
told to shutdown. These agents are subsequently removed from the master which 
causes tasks to be updated as TASK_LOST. Any agents from machines in 
maintenance are also prevented from registering with the master.“

 

Is there another way?

 

 

Cheers

 

Mike

 

Re: make slaves not getting tasks anymore

[Jeremy Olexa]

Hi Mike,


Yes, there is another way besides the maintenance primitives that aren't fully 
complete yet (IMO). If you wish to not schedule anymore jobs, you can remove 
that host from the whitelist on the masters. You might have to engineer this 
for your setup abit, but this is what we do:


1) All slaves are discovered and explicitly added to the whitelist

2) On demand (by the operator), a node is REMOVED from the whitelist for some 
time, currently we add the node back after a timeout of 1 hour

3) Wait for jobs to finish on that node, or send SIGUSR1 to mesos-slave process 
to force job termination


Of course, there is also satellite, which does all this for you :) 
https://github.com/twosigma/satellite/

Hope that helps,
-Jeremy



From: Mike Michel 
Sent: Wednesday, December 30, 2015 5:43 AM
To: user@mesos.apache.org
Subject: make slaves not getting tasks anymore


Hi,



i need to update slaves from time to time and looking for a way to take them 
out of the cluster but without killing the running tasks. I need to wait until 
all tasks are done and during this time no new tasks should be started on this 
slave. My first idea was to set a constraint "status:online" for every task i 
start and then change the attribute of the slave to "offline", restart slave 
process while executer still runs the tasks but it seems if you change the 
attributes of a slave it can not connect to the cluster without rm -rf /tmp 
before which will kill all tasks.



Also the maintenance mode seems not to be an option:



"When maintenance is triggered by the operator, all agents on the machine are 
told to shutdown. These agents are subsequently removed from the master which 
causes tasks to be updated as TASK_LOST. Any agents from machines in 
maintenance are also prevented from registering with the master."




Is there another way?





Cheers



Mike

Re: The issue of "Failed to shutdown socket with fd xx: Transport endpoint is not connected" on Mesos master

[Avinash Sridharan]

Thanks for the update Nan. k8s enabling firewall rules that would block
traffic to the master seems a bit odd. Looks like a bug to me, in the head
of the branch. If you are able to reproduce it consistently, could you file
an issue against kubernetes mesos.

regards,
Avinash

On Tue, Dec 29, 2015 at 11:04 PM, Nan Xiao  wrote:

> Hi Avinash,
>
> Thanks very much for your reply!
>
> The root cause has been found: the k8s server has enabled the iptables
> which blocks connection from
> Mesos master; after disable it, it works!
>
> Best Regards
> Nan Xiao
>
>
> On Wed, Dec 30, 2015 at 3:22 AM, Avinash Sridharan
>  wrote:
> > lsof command will show only actively opened file descriptors. So if you
> ran
> > the command after seeing the error logs in the master, most probably the
> > master had already closed this fd. Just throwing a few other things to
> look
> > at, that might give some more insights.
> >
> > * Run the "netstat -na" and netstat -nt" commands on the master and the
> > kubernetes master node to make sure that the master is listening to the
> > right port, and the k8s scheduler is trying to connect to the right port.
> > From the logs it does look like the master is receiving the registration
> > request, so there shouldn't be a network configuration issue here.
> > * Make sure there are no firewall rules getting turned on in your cluster
> > since it looks like the k8s scheduler is not able to connect to the
> master
> > (though it was able to register the first time).
> >
> > On Tue, Dec 29, 2015 at 1:37 AM, Nan Xiao 
> wrote:
> >>
> >> BTW, using "lsof" command finds there are only 16 file descriptors. I
> >> don't know why Mesos
> >> master try to close "fd 17".
> >> Best Regards
> >> Nan Xiao
> >>
> >>
> >> On Tue, Dec 29, 2015 at 11:32 AM, Nan Xiao 
> >> wrote:
> >> > Hi Klaus,
> >> >
> >> > Firstly, thanks very much for your answer!
> >> >
> >> > The km processes are all live:
> >> > root 129474 128024  2 22:26 pts/000:00:00 km apiserver
> >> > --address=15.242.100.60 --etcd-servers=http://15.242.100.60:4001
> >> > --service-cluster-ip-range=10.10.10.0/24 --port=
> >> > --cloud-provider=mesos --cloud-config=mesos-cloud.conf --secure-port=0
> >> > --v=1
> >> > root 129509 128024  2 22:26 pts/000:00:00 km
> >> > controller-manager --master=15.242.100.60: --cloud-provider=mesos
> >> > --cloud-config=./mesos-cloud.conf --v=1
> >> > root 129538 128024  0 22:26 pts/000:00:00 km scheduler
> >> > --address=15.242.100.60 --mesos-master=15.242.100.56:5050
> >> > --etcd-servers=http://15.242.100.60:4001 --mesos-user=root
> >> > --api-servers=15.242.100.60: --cluster-dns=10.10.10.10
> >> > --cluster-domain=cluster.local --v=2
> >> >
> >> > All the logs are also seem OK, except the logs from scheduler.log:
> >> > ..
> >> > I1228 22:26:37.883092  129538 messenger.go:381] Receiving message
> >> > mesos.internal.InternalMasterChangeDetected from
> >> > scheduler(1)@15.242.100.60:33077
> >> > I1228 22:26:37.883225  129538 scheduler.go:374] New master
> >> > master@15.242.100.56:5050 detected
> >> > I1228 22:26:37.883268  129538 scheduler.go:435] No credentials were
> >> > provided. Attempting to register scheduler without authentication.
> >> > I1228 22:26:37.883356  129538 scheduler.go:928] Registering with
> >> > master: master@15.242.100.56:5050
> >> > I1228 22:26:37.883460  129538 messenger.go:187] Sending message
> >> > mesos.internal.RegisterFrameworkMessage to master@15.242.100.56:5050
> >> > I1228 22:26:37.883504  129538 scheduler.go:881] will retry
> >> > registration in 1.209320575s if necessary
> >> > I1228 22:26:37.883758  129538 http_transporter.go:193] Sending message
> >> > to master@15.242.100.56:5050 via http
> >> > I1228 22:26:37.883873  129538 http_transporter.go:587] libproc target
> >> > URL
> >> >
> http://15.242.100.56:5050/master/mesos.internal.RegisterFrameworkMessage
> >> > I1228 22:26:39.093560  129538 scheduler.go:928] Registering with
> >> > master: master@15.242.100.56:5050
> >> > I1228 22:26:39.093659  129538 messenger.go:187] Sending message
> >> > mesos.internal.RegisterFrameworkMessage to master@15.242.100.56:5050
> >> > I1228 22:26:39.093702  129538 scheduler.go:881] will retry
> >> > registration in 3.762036352s if necessary
> >> > I1228 22:26:39.093765  129538 http_transporter.go:193] Sending message
> >> > to master@15.242.100.56:5050 via http
> >> > I1228 22:26:39.093847  129538 http_transporter.go:587] libproc target
> >> > URL
> >> >
> http://15.242.100.56:5050/master/mesos.internal.RegisterFrameworkMessage
> >> > ..
> >> >
> >> > From the log, the Mesos master rejected the k8s's registeration, and
> >> > k8s retry constantly.
> >> >
> >> > Have you met this issue before? Thanks very much in advance!
> >> > Best Regards
> >> > Nan Xiao
> >> >
> >> >
> >> > On Mon, Dec 28, 2015 at 7:26 PM, Klaus Ma 
> >> > wrote:
> >> >> It seems Kubernetes is down; would you help to check kubernetes's
> >> >> status
> >> >> (km)?
> >> >>
> 

Re: make slaves not getting tasks anymore

[Shuai Lin]

>
> I need to wait until all tasks are done and during this time no new tasks
> should be started on this slave


This is  exactly what maintenance mode is designed for. But to achieve
this, it requires the cooperation of the framework. When the operator adds
a maintenance schedule for a slave, mesos master would first send "inverse
offers" to all frameworks that have tasks running on that slave, and the
frameworks are "assumed to" move the tasks away to other slaves.

But the framework can ignore the inverse offers as well, for example, I
can't find any code to handle it in marathon code.




> Also the maintenance mode seems not to be an option:

When maintenance is triggered by the operator, all agents on the machine
> are told to shutdown


Be aware that the maintenance process is a two-phase process:

- the first step is "adding the maintenance schedule", the operator tells
master "I would take slaveX down for maintenance in 1 hour, please ask the
frameworks to move their tasks to other slaves", as I described above
- the second step is "starting the maintenance", the operator tells the
master "I'm taking this slave down RIGHT NOW". The master would kill all
tasks on that slave and asks the mesos-slave process to exit, as described
in the paragrah you quoted in the original mesasge.

In a word, it mostly depends on the frameworks you use.


On Wed, Dec 30, 2015 at 7:43 PM, Mike Michel  wrote:

> Hi,
>
>
>
> i need to update slaves from time to time and looking for a way to take
> them out of the cluster but without killing the running tasks. I need to
> wait until all tasks are done and during this time no new tasks should be
> started on this slave. My first idea was to set a constraint
> „status:online“ for every task i start and then change the attribute of the
> slave to „offline“, restart slave process while executer still runs the
> tasks but it seems if you change the attributes of a slave it can not
> connect to the cluster without rm -rf /tmp before which will kill all tasks.
>
>
>
> Also the maintenance mode seems not to be an option:
>
>
>
> „When maintenance is triggered by the operator, all agents on the machine
> are told to shutdown. These agents are subsequently removed from the master
> which causes tasks to be updated as TASK_LOST. Any agents from machines
> in maintenance are also prevented from registering with the master.“
>
>
>
> Is there another way?
>
>
>
>
>
> Cheers
>
>
>
> Mike
>

Re: make slaves not getting tasks anymore

[Klaus Ma]

Hi Mike,

Which framework are you using? How about Maintenance's scheduling feature?
My understanding is that framework show not dispatch task to the
maintenance agent; so Operator can wait for all tasks finished before
taking any action.

For "When maintenance is triggered by the operator", it's used when
there're some tasks took too long time to finish; so Operator can task
action to shut them down.

For the agent restart with new attributes, there's a JIRA (MESOS-1739)
about it.


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Wed, Dec 30, 2015 at 7:43 PM, Mike Michel  wrote:

> Hi,
>
>
>
> i need to update slaves from time to time and looking for a way to take
> them out of the cluster but without killing the running tasks. I need to
> wait until all tasks are done and during this time no new tasks should be
> started on this slave. My first idea was to set a constraint
> „status:online“ for every task i start and then change the attribute of the
> slave to „offline“, restart slave process while executer still runs the
> tasks but it seems if you change the attributes of a slave it can not
> connect to the cluster without rm -rf /tmp before which will kill all tasks.
>
>
>
> Also the maintenance mode seems not to be an option:
>
>
>
> „When maintenance is triggered by the operator, all agents on the machine
> are told to shutdown. These agents are subsequently removed from the master
> which causes tasks to be updated as TASK_LOST. Any agents from machines
> in maintenance are also prevented from registering with the master.“
>
>
>
> Is there another way?
>
>
>
>
>
> Cheers
>
>
>
> Mike
>

Re: make slaves not getting tasks anymore

[Dick Davies]

It sounds like you want to use checkpointing, that should keep the
tasks alive as you update
the mesos slave process itself.

On 30 December 2015 at 11:43, Mike Michel  wrote:
> Hi,
>
>
>
> i need to update slaves from time to time and looking for a way to take them
> out of the cluster but without killing the running tasks. I need to wait
> until all tasks are done and during this time no new tasks should be started
> on this slave. My first idea was to set a constraint „status:online“ for
> every task i start and then change the attribute of the slave to „offline“,
> restart slave process while executer still runs the tasks but it seems if
> you change the attributes of a slave it can not connect to the cluster
> without rm -rf /tmp before which will kill all tasks.
>
>
>
> Also the maintenance mode seems not to be an option:
>
>
>
> „When maintenance is triggered by the operator, all agents on the machine
> are told to shutdown. These agents are subsequently removed from the master
> which causes tasks to be updated as TASK_LOST. Any agents from machines in
> maintenance are also prevented from registering with the master.“
>
>
>
> Is there another way?
>
>
>
>
>
> Cheers
>
>
>
> Mike

make slaves not getting tasks anymore

[Mike Michel]

Hi,

 

i need to update slaves from time to time and looking for a way to take them
out of the cluster but without killing the running tasks. I need to wait
until all tasks are done and during this time no new tasks should be started
on this slave. My first idea was to set a constraint "status:online" for
every task i start and then change the attribute of the slave to "offline",
restart slave process while executer still runs the tasks but it seems if
you change the attributes of a slave it can not connect to the cluster
without rm -rf /tmp before which will kill all tasks.

 

Also the maintenance mode seems not to be an option:

 

"When maintenance is triggered by the operator, all agents on the machine
are told to shutdown. These agents are subsequently removed from the master
which causes tasks to be updated as TASK_LOST. Any agents from machines in
maintenance are also prevented from registering with the master."



 

Is there another way?

 

 

Cheers

 

Mike

Re: How can mesos print logs from VLOG function?

[Nan Xiao]

Hi Marco,

Yes, it worked, thanks very much!
Best Regards
Nan Xiao


On Wed, Dec 30, 2015 at 4:55 PM, Marco Massenzio  wrote:
> Mesos uses Google Logging[0] and, according to the documentation there, the
> VLOG(n) calls are only logged if a variable GLOG_v=m (where n > m) is
> configured when running Mesos (the other suggested way, using --v=m won't
> work for mesos).
>
> Having said that, I have recently been unable to make this work - so there
> may be some other trickery at work.
>
> [0] https://google-glog.googlecode.com/svn/trunk/doc/glog.html
>
> --
> Marco Massenzio
> http://codetrips.com
>
> On Wed, Dec 30, 2015 at 12:30 AM, Nan Xiao  wrote:
>>
>> Hi all,
>>
>> I want mesos prints logs from VLOG function:
>>
>> VLOG(1) << "Executor started at: " << self()
>> << " with pid " << getpid();
>>
>> But from mesos help:
>>
>> $ sudo ./bin/mesos-master.sh --help | grep -i LOG
>>   --external_log_file=VALUESpecified the externally
>> managed log file. This file will be
>>stderr logging as the log
>> file is otherwise unknown to Mesos.
>>   --[no-]initialize_driver_logging Whether to automatically
>> initialize Google logging of scheduler
>>   --[no-]log_auto_initialize   Whether to automatically
>> initialize the replicated log used for the
>>registry. If this is set to
>> false, the log has to be manually
>>   --log_dir=VALUE  Directory path to put log
>> files (no default, nothing
>>does not affect logging to
>> stderr).
>>NOTE: 3rd party log
>> messages (e.g. ZooKeeper) are
>>   --logbufsecs=VALUE   How many seconds to buffer
>> log messages for (default: 0)
>>   --logging_level=VALUELog message at or above
>> this level; possible values:
>>will affect just the logs
>> from log_dir (if specified) (default: INFO)
>>   --[no-]quiet Disable logging to stderr
>> (default: false)
>>   --quorum=VALUE   The size of the quorum of
>> replicas when using 'replicated_log' based
>>available options are
>> 'replicated_log', 'in_memory' (for testing). (default: replicated_log)
>>
>> I can't find related configurations.
>>
>> So how can mesos print  logs from VLOG function? Thanks in advance!
>>
>> Best Regards
>> Nan Xiao
>
>

Re: How can mesos print logs from VLOG function?

[Marco Massenzio]

Mesos uses Google Logging[0] and, according to the documentation there, the
VLOG(n) calls are only logged if a variable GLOG_v=m (where n > m) is
configured when running Mesos (the other suggested way, using --v=m won't
work for mesos).

Having said that, I have recently been unable to make this work - so there
may be some other trickery at work.

[0] https://google-glog.googlecode.com/svn/trunk/doc/glog.html

-- 
*Marco Massenzio*
http://codetrips.com

On Wed, Dec 30, 2015 at 12:30 AM, Nan Xiao  wrote:

> Hi all,
>
> I want mesos prints logs from VLOG function:
>
> VLOG(1) << "Executor started at: " << self()
> << " with pid " << getpid();
>
> But from mesos help:
>
> $ sudo ./bin/mesos-master.sh --help | grep -i LOG
>   --external_log_file=VALUESpecified the externally
> managed log file. This file will be
>stderr logging as the log
> file is otherwise unknown to Mesos.
>   --[no-]initialize_driver_logging Whether to automatically
> initialize Google logging of scheduler
>   --[no-]log_auto_initialize   Whether to automatically
> initialize the replicated log used for the
>registry. If this is set to
> false, the log has to be manually
>   --log_dir=VALUE  Directory path to put log
> files (no default, nothing
>does not affect logging to
> stderr).
>NOTE: 3rd party log
> messages (e.g. ZooKeeper) are
>   --logbufsecs=VALUE   How many seconds to buffer
> log messages for (default: 0)
>   --logging_level=VALUELog message at or above
> this level; possible values:
>will affect just the logs
> from log_dir (if specified) (default: INFO)
>   --[no-]quiet Disable logging to stderr
> (default: false)
>   --quorum=VALUE   The size of the quorum of
> replicas when using 'replicated_log' based
>available options are
> 'replicated_log', 'in_memory' (for testing). (default: replicated_log)
>
> I can't find related configurations.
>
> So how can mesos print  logs from VLOG function? Thanks in advance!
>
> Best Regards
> Nan Xiao
>

How can mesos print logs from VLOG function?

[Nan Xiao]

Hi all,

I want mesos prints logs from VLOG function:

VLOG(1) << "Executor started at: " << self()
<< " with pid " << getpid();

But from mesos help:

$ sudo ./bin/mesos-master.sh --help | grep -i LOG
  --external_log_file=VALUESpecified the externally
managed log file. This file will be
   stderr logging as the log
file is otherwise unknown to Mesos.
  --[no-]initialize_driver_logging Whether to automatically
initialize Google logging of scheduler
  --[no-]log_auto_initialize   Whether to automatically
initialize the replicated log used for the
   registry. If this is set to
false, the log has to be manually
  --log_dir=VALUE  Directory path to put log
files (no default, nothing
   does not affect logging to stderr).
   NOTE: 3rd party log
messages (e.g. ZooKeeper) are
  --logbufsecs=VALUE   How many seconds to buffer
log messages for (default: 0)
  --logging_level=VALUELog message at or above
this level; possible values:
   will affect just the logs
from log_dir (if specified) (default: INFO)
  --[no-]quiet Disable logging to stderr
(default: false)
  --quorum=VALUE   The size of the quorum of
replicas when using 'replicated_log' based
   available options are
'replicated_log', 'in_memory' (for testing). (default: replicated_log)

I can't find related configurations.

So how can mesos print  logs from VLOG function? Thanks in advance!

Best Regards
Nan Xiao