Olivier, on a side note, it's great to see that you are playing with the new HTTP API in python! I briefly looked at your linked code and it looks like you are mixing the business logic of your application and the Mesos API interaction in the same file. It would be great if (at some point) you can extract the Mesos API interaction into a python library that can be used by other frameworks. See other libraries (C++ <https://github.com/apache/mesos/blob/master/include/mesos/v1/scheduler.hpp>, Java <https://github.com/mesosphere/mesos-rxjava>, Go <https://github.com/mesos/mesos-go>) for inspiration.
On Tue, Jun 7, 2016 at 11:46 AM, Anand Mazumdar <[email protected]> wrote: > Olivier, > > You are missing the “task_infos” key in your “ACCEPT” call. The master > treats “Accept” operations with no launch tasks as declining offers > implicitly. I would file a followup JIRA to ensure this is logged on the > master (if not so). > > An example correct JSON: > https://gist.github.com/hatred/7325d8a4afde607ecc0f376ab62d60eb < > https://gist.github.com/hatred/7325d8a4afde607ecc0f376ab62d60eb> > > -anand > > > On Jun 7, 2016, at 8:38 AM, Olivier Sallou <[email protected]> > wrote: > > > > > > > > On 06/07/2016 04:53 PM, Guangya Liu wrote: > >> So how many agent nodes are there in your cluster? If you continue > >> receiving offer but without getting UPDATE message, then it may be > caused > >> by that your task definition and the framework continually decline > offer. > > I have only one node (master/slave), for development. It worked fine > > with the python API. > > we see on master that it received the ACCEPT, and no DECLINE. However, > > as I receive no UPDATE, I suppose that mesos "drops" the ACCEPT (wrong > > task definition maybe), and sends new offers several seconds after I > > sent the ACCEPT. > >> > >> Can you please share your framework code here for the logic of "Event:: > >> OFFERS"? > > Code is available here: > > > > > https://bitbucket.org/osallou/go-docker/src/b1948063fb7f68fbc77f5de6b473d832a7dd36af/plugins/mesos.py?at=master&fileviewer=file-view-default > < > https://bitbucket.org/osallou/go-docker/src/b1948063fb7f68fbc77f5de6b473d832a7dd36af/plugins/mesos.py?at=master&fileviewer=file-view-default > > > > > > in method run of MesosThread, line 613 > > > > Code is a little complex, as it is a port of existing code using mesos > > python lib. > > > > Code related to HTTP is in development, so there may be further errors, > > but registration is fine as well as offer messages. > > > > I have added locally a debug print to show any message received by mesos > > (in case I would have received an other message indicating an error), > > but I received no other than offer and heartbeats. > > > > If Mesos see the ACCEPT message as it appears in logs, that it should > > either reject it (with a different status code than 202) or send an > > UPDATE error message if there is an error with my task definition. > > > > Olivier > >> > >> Thanks, > >> > >> Guangya > >> > >> On Tue, Jun 7, 2016 at 8:29 PM, Olivier Sallou <[email protected] > > > >> wrote: > >> > >>> > >>> On 06/07/2016 01:59 PM, Guangya Liu wrote: > >>>> I can see that your framework is now holding the offer, how did you > >>> launch > >>>> task? > >>> I execute an HTTP POST request in Python with json content-type: > >>> > >>> {'type': 'ACCEPT', > >>> 'framework_id': {'value': > u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, > >>> 'accept': { > >>> 'operations': [ > >>> {'type': 'LAUNCH', > >>> 'launch': {'container': { > >>> 'docker': {'image': u'centos:latest', > >>> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, > >>> 'type': 1, > >>> 'volumes': [ > >>> {'host_path': u'/a/b', 'container_path': > >>> u'/mnt/home', 'mode': 1}, > >>> {'host_path': u'/a/b/c', 'container_path': > >>> u'/mnt/go-docker', 'mode': 1}, > >>> {'host_path': u'/b/c/d', 'container_path': > >>> u'/mnt/god-data', 'mode': 2} > >>> ] > >>> }, > >>> 'name': u'testr', > >>> 'task_id': {'value': '128'}, > >>> 'command': {'uris': [{'value': > >>> u'/home/osallou/docker.tar.gz'}], 'value': > u'/mnt/go-docker/wrapper.sh'}, > >>> 'slave_id': {'value': > >>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, > >>> 'resources': [ > >>> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, > >>> {'scalar': {'value': 2000}, 'type': 0, 'name': 'mem'} > >>> ] > >>> } # end launch > >>> } # end operation > >>> ], > >>> 'offer_ids': [{'value': > u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] > >>> } > >>> } > >>> > >>> We can see that Mesos received the ACCEPT: > >>> > >>> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call for > >>> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave > >>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 > >>> (tifenn.irisa.fr) for framework > >>> > >>> > >>> and I continue to receive new offers, so "connection" is OK. I should > >>> receive an UPDATE message even if there is an error, but I receive none > >>> (I track/log all messages received, whatever the type). > >>> > >>> Olivier > >>> > >>>> Perhaps you can take a look at > >>>> https://github.com/apache/mesos/blob/master/src/cli/execute.cpp#L311 > >>> which > >>>> is an example framework using HTTP API > >>>> > >>>> Thanks, > >>>> > >>>> Guangya > >>>> > >>>> On Tue, Jun 7, 2016 at 7:19 PM, Olivier Sallou < > [email protected]> > >>>> wrote: > >>>> > >>>>> On 06/07/2016 12:25 PM, Guangya Liu wrote: > >>>>>> Olivier, > >>>>>> > >>>>>> For such case, seems there is sth wrong with your framework? can you > >>>>> please > >>>>>> run the following two commands and check the output? > >>>>> I don't think it is a framework issue, I receive offers, heartbeats > >>> etc... > >>>>> It is only at task creation step, when I have no rejection nor update > >>>>> message. > >>>>> > >>>>> It could be (certainly) an issue with the json task message I sent in > >>>>> the ACCEPT, but as there is no error, I have no way to understand > what's > >>>>> wrong with it. > >>>>>> curl "http://<mesos-master>:5050/master/frameworks" > 2>/dev/null|python > >>>>> -m > >>>>>> json.tool > >>>>> { > >>>>> "completed_frameworks": [], > >>>>> "frameworks": [ > >>>>> { > >>>>> "active": true, > >>>>> "capabilities": [], > >>>>> "checkpoint": false, > >>>>> "completed_tasks": [], > >>>>> "executors": [], > >>>>> "failover_timeout": 0.0, > >>>>> "hostname": "", > >>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", > >>>>> "name": "GoDocker HTTP Framework", > >>>>> "offered_resources": { > >>>>> "cpus": 4.0, > >>>>> "disk": 459470.0, > >>>>> "mem": 14898.0, > >>>>> "ports": "[31000-32000]" > >>>>> }, > >>>>> "offers": [ > >>>>> { > >>>>> "framework_id": > >>>>> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", > >>>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956-O0", > >>>>> "resources": { > >>>>> "cpus": 4.0, > >>>>> "disk": 459470.0, > >>>>> "mem": 14898.0, > >>>>> "ports": "[31000-32000]" > >>>>> }, > >>>>> "slave_id": > >>> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0" > >>>>> } > >>>>> ], > >>>>> "registered_time": 1465298174.2483, > >>>>> "resources": { > >>>>> "cpus": 4.0, > >>>>> "disk": 459470.0, > >>>>> "mem": 14898.0, > >>>>> "ports": "[31000-32000]" > >>>>> }, > >>>>> "role": "*", > >>>>> "tasks": [], > >>>>> "unregistered_time": 0.0, > >>>>> "used_resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "user": "godocker_http_test", > >>>>> "webui_url": "" > >>>>> } > >>>>> ], > >>>>> "unregistered_frameworks": [] > >>>>> } > >>>>> > >>>>> > >>>>>> curl "http://<mesos-master>:5050/master/state" 2>/dev/null|python > -m > >>>>>> json.tool > >>>>> { > >>>>> "activated_slaves": 1.0, > >>>>> "build_date": "2016-04-14 15:44:54", > >>>>> "build_time": 1460648694.0, > >>>>> "build_user": "root", > >>>>> "completed_frameworks": [], > >>>>> "deactivated_slaves": 0.0, > >>>>> "elected_time": 1465298164.01165, > >>>>> "flags": { > >>>>> "allocation_interval": "1secs", > >>>>> "allocator": "HierarchicalDRF", > >>>>> "authenticate": "false", > >>>>> "authenticate_http": "false", > >>>>> "authenticate_slaves": "false", > >>>>> "authenticators": "crammd5", > >>>>> "authorizers": "local", > >>>>> "framework_sorter": "drf", > >>>>> "help": "false", > >>>>> "hostname_lookup": "true", > >>>>> "http_authenticators": "basic", > >>>>> "initialize_driver_logging": "true", > >>>>> "log_auto_initialize": "true", > >>>>> "log_dir": "/var/log/mesos", > >>>>> "logbufsecs": "0", > >>>>> "logging_level": "INFO", > >>>>> "max_completed_frameworks": "50", > >>>>> "max_completed_tasks_per_framework": "1000", > >>>>> "max_slave_ping_timeouts": "5", > >>>>> "port": "5050", > >>>>> "quiet": "false", > >>>>> "quorum": "1", > >>>>> "recovery_slave_removal_limit": "100%", > >>>>> "registry": "replicated_log", > >>>>> "registry_fetch_timeout": "1mins", > >>>>> "registry_store_timeout": "20secs", > >>>>> "registry_strict": "false", > >>>>> "root_submissions": "true", > >>>>> "slave_ping_timeout": "15secs", > >>>>> "slave_reregister_timeout": "10mins", > >>>>> "user_sorter": "drf", > >>>>> "version": "false", > >>>>> "webui_dir": "/usr/share/mesos/webui", > >>>>> "work_dir": "/var/lib/mesos", > >>>>> "zk": "zk://localhost:2181/mesos", > >>>>> "zk_session_timeout": "10secs" > >>>>> }, > >>>>> "frameworks": [ > >>>>> { > >>>>> "active": true, > >>>>> "capabilities": [], > >>>>> "checkpoint": false, > >>>>> "completed_tasks": [], > >>>>> "executors": [], > >>>>> "failover_timeout": 0.0, > >>>>> "hostname": "", > >>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", > >>>>> "name": "GoDocker HTTP Framework", > >>>>> "offered_resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "offers": [], > >>>>> "registered_time": 1465298174.2483, > >>>>> "resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "role": "*", > >>>>> "tasks": [], > >>>>> "unregistered_time": 0.0, > >>>>> "used_resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "user": "godocker_http_test", > >>>>> "webui_url": "" > >>>>> } > >>>>> ], > >>>>> "git_sha": "555db235a34afbb9fb49940376cc33a66f1f85f0", > >>>>> "git_tag": "0.28.1", > >>>>> "hostname": "tifenn.irisa.fr", > >>>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956", > >>>>> "leader": "[email protected]:5050", > >>>>> "log_dir": "/var/log/mesos", > >>>>> "orphan_tasks": [], > >>>>> "pid": "[email protected]:5050", > >>>>> "slaves": [ > >>>>> { > >>>>> "active": true, > >>>>> "attributes": { > >>>>> "hostname": "127.0.0.1" > >>>>> }, > >>>>> "hostname": "tifenn.irisa.fr", > >>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0", > >>>>> "offered_resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "pid": "slave(1)@127.0.1.1:5051", > >>>>> "registered_time": 1465298164.37517, > >>>>> "reregistered_time": 1465298164.37526, > >>>>> "reserved_resources": {}, > >>>>> "resources": { > >>>>> "cpus": 4.0, > >>>>> "disk": 459470.0, > >>>>> "mem": 14898.0, > >>>>> "ports": "[31000-32000]" > >>>>> }, > >>>>> "unreserved_resources": { > >>>>> "cpus": 4.0, > >>>>> "disk": 459470.0, > >>>>> "mem": 14898.0, > >>>>> "ports": "[31000-32000]" > >>>>> }, > >>>>> "used_resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "version": "0.28.1" > >>>>> } > >>>>> ], > >>>>> "start_time": 1465298159.26321, > >>>>> "unregistered_frameworks": [], > >>>>> "version": "0.28.1" > >>>>> } > >>>>> > >>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Guangya > >>>>>> > >>>>>> On Tue, Jun 7, 2016 at 6:04 PM, Olivier Sallou < > >>> [email protected]> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi, > >>>>>>> I am trying to switch from Python to HTTP API. I use mesos 0.28.1 > >>>>>>> > >>>>>>> I could create framework to register, receive offers etc... but I > >>> have > >>>>>>> an issue accepting offers. > >>>>>>> > >>>>>>> I send my ACCEPT message but I do not receive any UPDATE message, > only > >>>>>>> new offers and hearbeat messages. > >>>>>>> > >>>>>>> On mesos master logs I see: > >>>>>>> > >>>>>>> I0607 11:45:15.873184 14896 http.cpp:312] HTTP POST for > >>>>>>> /master/api/v1/scheduler from 127.0.0.1:38298 with > >>>>>>> User-Agent='python-requests/2.9.1' > >>>>>>> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call > >>> for > >>>>>>> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave > >>>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 > >>>>>>> (tifenn.irisa.fr) for framework > >>>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020 (GoDocker HTTP Framework) > >>>>>>> > >>>>>>> There is a "Processing ACCEPT" and no error, but my task is not > ran on > >>>>>>> mesos. > >>>>>>> No error on slave either. > >>>>>>> > >>>>>>> Response code to my ACCEPT is 202 as expected. > >>>>>>> > >>>>>>> Here is my HTTP json message: > >>>>>>> > >>>>>>> {'type': 'ACCEPT', > >>>>>>> 'framework_id': {'value': > >>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, > >>>>>>> 'accept': { > >>>>>>> 'operations': [ > >>>>>>> {'type': 'LAUNCH', > >>>>>>> 'launch': {'container': { > >>>>>>> 'docker': {'image': u'centos:latest', > >>>>>>> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, > >>>>>>> 'type': 1, > >>>>>>> 'volumes': [ > >>>>>>> {'host_path': u'/a/b', 'container_path': > >>>>>>> u'/mnt/home', 'mode': 1}, > >>>>>>> {'host_path': u'/a/b/c', 'container_path': > >>>>>>> u'/mnt/go-docker', 'mode': 1}, > >>>>>>> {'host_path': u'/b/c/d', 'container_path': > >>>>>>> u'/mnt/god-data', 'mode': 2} > >>>>>>> ] > >>>>>>> }, > >>>>>>> 'name': u'testr', > >>>>>>> 'task_id': {'value': '128'}, > >>>>>>> 'command': {'uris': [{'value': > >>>>>>> u'/home/osallou/docker.tar.gz'}], 'value': > >>>>> u'/mnt/go-docker/wrapper.sh'}, > >>>>>>> 'slave_id': {'value': > >>>>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, > >>>>>>> 'resources': [ > >>>>>>> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, > >>>>>>> {'scalar': {'value': 2000}, 'type': 0, 'name': > 'mem'} > >>>>>>> ] > >>>>>>> } # end launch > >>>>>>> } # end operation > >>>>>>> ], > >>>>>>> 'offer_ids': [{'value': > >>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] > >>>>>>> } > >>>>>>> } > >>>>>>> > >>>>>>> There could be an issue with my task definition, but as no error is > >>>>>>> raised and I receive no UPDATE error message..... > >>>>>>> > >>>>>>> Any hint on how to debug this? > >>>>>>> > >>>>>>> Thanks > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Olivier Sallou > >>>>>>> IRISA / University of Rennes 1 > >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE > >>>>>>> Tel: 02.99.84.71.95 > >>>>>>> > >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) > >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D > 8438 > >>>>>>> > >>>>>>> > >>>>> -- > >>>>> Olivier Sallou > >>>>> IRISA / University of Rennes 1 > >>>>> Campus de Beaulieu, 35000 RENNES - FRANCE > >>>>> Tel: 02.99.84.71.95 > >>>>> > >>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) > >>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > >>>>> > >>>>> > >>> -- > >>> Olivier Sallou > >>> IRISA / University of Rennes 1 > >>> Campus de Beaulieu, 35000 RENNES - FRANCE > >>> Tel: 02.99.84.71.95 > >>> > >>> gpg key id: 4096R/326D8438 (keyring.debian.org) > >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > >>> > >>> > >>> > > > > -- > > Olivier Sallou > > IRISA / University of Rennes 1 > > Campus de Beaulieu, 35000 RENNES - FRANCE > > Tel: 02.99.84.71.95 > > > > gpg key id: 4096R/326D8438 (keyring.debian.org < > http://keyring.debian.org/>) > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > >
