Olivier, You are missing the “task_infos” key in your “ACCEPT” call. The master treats “Accept” operations with no launch tasks as declining offers implicitly. I would file a followup JIRA to ensure this is logged on the master (if not so).
An example correct JSON: https://gist.github.com/hatred/7325d8a4afde607ecc0f376ab62d60eb <https://gist.github.com/hatred/7325d8a4afde607ecc0f376ab62d60eb> -anand > On Jun 7, 2016, at 8:38 AM, Olivier Sallou <[email protected]> wrote: > > > > On 06/07/2016 04:53 PM, Guangya Liu wrote: >> So how many agent nodes are there in your cluster? If you continue >> receiving offer but without getting UPDATE message, then it may be caused >> by that your task definition and the framework continually decline offer. > I have only one node (master/slave), for development. It worked fine > with the python API. > we see on master that it received the ACCEPT, and no DECLINE. However, > as I receive no UPDATE, I suppose that mesos "drops" the ACCEPT (wrong > task definition maybe), and sends new offers several seconds after I > sent the ACCEPT. >> >> Can you please share your framework code here for the logic of "Event:: >> OFFERS"? > Code is available here: > > https://bitbucket.org/osallou/go-docker/src/b1948063fb7f68fbc77f5de6b473d832a7dd36af/plugins/mesos.py?at=master&fileviewer=file-view-default > > <https://bitbucket.org/osallou/go-docker/src/b1948063fb7f68fbc77f5de6b473d832a7dd36af/plugins/mesos.py?at=master&fileviewer=file-view-default> > > in method run of MesosThread, line 613 > > Code is a little complex, as it is a port of existing code using mesos > python lib. > > Code related to HTTP is in development, so there may be further errors, > but registration is fine as well as offer messages. > > I have added locally a debug print to show any message received by mesos > (in case I would have received an other message indicating an error), > but I received no other than offer and heartbeats. > > If Mesos see the ACCEPT message as it appears in logs, that it should > either reject it (with a different status code than 202) or send an > UPDATE error message if there is an error with my task definition. > > Olivier >> >> Thanks, >> >> Guangya >> >> On Tue, Jun 7, 2016 at 8:29 PM, Olivier Sallou <[email protected]> >> wrote: >> >>> >>> On 06/07/2016 01:59 PM, Guangya Liu wrote: >>>> I can see that your framework is now holding the offer, how did you >>> launch >>>> task? >>> I execute an HTTP POST request in Python with json content-type: >>> >>> {'type': 'ACCEPT', >>> 'framework_id': {'value': u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, >>> 'accept': { >>> 'operations': [ >>> {'type': 'LAUNCH', >>> 'launch': {'container': { >>> 'docker': {'image': u'centos:latest', >>> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, >>> 'type': 1, >>> 'volumes': [ >>> {'host_path': u'/a/b', 'container_path': >>> u'/mnt/home', 'mode': 1}, >>> {'host_path': u'/a/b/c', 'container_path': >>> u'/mnt/go-docker', 'mode': 1}, >>> {'host_path': u'/b/c/d', 'container_path': >>> u'/mnt/god-data', 'mode': 2} >>> ] >>> }, >>> 'name': u'testr', >>> 'task_id': {'value': '128'}, >>> 'command': {'uris': [{'value': >>> u'/home/osallou/docker.tar.gz'}], 'value': u'/mnt/go-docker/wrapper.sh'}, >>> 'slave_id': {'value': >>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, >>> 'resources': [ >>> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, >>> {'scalar': {'value': 2000}, 'type': 0, 'name': 'mem'} >>> ] >>> } # end launch >>> } # end operation >>> ], >>> 'offer_ids': [{'value': u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] >>> } >>> } >>> >>> We can see that Mesos received the ACCEPT: >>> >>> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call for >>> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave >>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 >>> (tifenn.irisa.fr) for framework >>> >>> >>> and I continue to receive new offers, so "connection" is OK. I should >>> receive an UPDATE message even if there is an error, but I receive none >>> (I track/log all messages received, whatever the type). >>> >>> Olivier >>> >>>> Perhaps you can take a look at >>>> https://github.com/apache/mesos/blob/master/src/cli/execute.cpp#L311 >>> which >>>> is an example framework using HTTP API >>>> >>>> Thanks, >>>> >>>> Guangya >>>> >>>> On Tue, Jun 7, 2016 at 7:19 PM, Olivier Sallou <[email protected]> >>>> wrote: >>>> >>>>> On 06/07/2016 12:25 PM, Guangya Liu wrote: >>>>>> Olivier, >>>>>> >>>>>> For such case, seems there is sth wrong with your framework? can you >>>>> please >>>>>> run the following two commands and check the output? >>>>> I don't think it is a framework issue, I receive offers, heartbeats >>> etc... >>>>> It is only at task creation step, when I have no rejection nor update >>>>> message. >>>>> >>>>> It could be (certainly) an issue with the json task message I sent in >>>>> the ACCEPT, but as there is no error, I have no way to understand what's >>>>> wrong with it. >>>>>> curl "http://<mesos-master>:5050/master/frameworks" 2>/dev/null|python >>>>> -m >>>>>> json.tool >>>>> { >>>>> "completed_frameworks": [], >>>>> "frameworks": [ >>>>> { >>>>> "active": true, >>>>> "capabilities": [], >>>>> "checkpoint": false, >>>>> "completed_tasks": [], >>>>> "executors": [], >>>>> "failover_timeout": 0.0, >>>>> "hostname": "", >>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>>> "name": "GoDocker HTTP Framework", >>>>> "offered_resources": { >>>>> "cpus": 4.0, >>>>> "disk": 459470.0, >>>>> "mem": 14898.0, >>>>> "ports": "[31000-32000]" >>>>> }, >>>>> "offers": [ >>>>> { >>>>> "framework_id": >>>>> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956-O0", >>>>> "resources": { >>>>> "cpus": 4.0, >>>>> "disk": 459470.0, >>>>> "mem": 14898.0, >>>>> "ports": "[31000-32000]" >>>>> }, >>>>> "slave_id": >>> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0" >>>>> } >>>>> ], >>>>> "registered_time": 1465298174.2483, >>>>> "resources": { >>>>> "cpus": 4.0, >>>>> "disk": 459470.0, >>>>> "mem": 14898.0, >>>>> "ports": "[31000-32000]" >>>>> }, >>>>> "role": "*", >>>>> "tasks": [], >>>>> "unregistered_time": 0.0, >>>>> "used_resources": { >>>>> "cpus": 0.0, >>>>> "disk": 0.0, >>>>> "mem": 0.0 >>>>> }, >>>>> "user": "godocker_http_test", >>>>> "webui_url": "" >>>>> } >>>>> ], >>>>> "unregistered_frameworks": [] >>>>> } >>>>> >>>>> >>>>>> curl "http://<mesos-master>:5050/master/state" 2>/dev/null|python -m >>>>>> json.tool >>>>> { >>>>> "activated_slaves": 1.0, >>>>> "build_date": "2016-04-14 15:44:54", >>>>> "build_time": 1460648694.0, >>>>> "build_user": "root", >>>>> "completed_frameworks": [], >>>>> "deactivated_slaves": 0.0, >>>>> "elected_time": 1465298164.01165, >>>>> "flags": { >>>>> "allocation_interval": "1secs", >>>>> "allocator": "HierarchicalDRF", >>>>> "authenticate": "false", >>>>> "authenticate_http": "false", >>>>> "authenticate_slaves": "false", >>>>> "authenticators": "crammd5", >>>>> "authorizers": "local", >>>>> "framework_sorter": "drf", >>>>> "help": "false", >>>>> "hostname_lookup": "true", >>>>> "http_authenticators": "basic", >>>>> "initialize_driver_logging": "true", >>>>> "log_auto_initialize": "true", >>>>> "log_dir": "/var/log/mesos", >>>>> "logbufsecs": "0", >>>>> "logging_level": "INFO", >>>>> "max_completed_frameworks": "50", >>>>> "max_completed_tasks_per_framework": "1000", >>>>> "max_slave_ping_timeouts": "5", >>>>> "port": "5050", >>>>> "quiet": "false", >>>>> "quorum": "1", >>>>> "recovery_slave_removal_limit": "100%", >>>>> "registry": "replicated_log", >>>>> "registry_fetch_timeout": "1mins", >>>>> "registry_store_timeout": "20secs", >>>>> "registry_strict": "false", >>>>> "root_submissions": "true", >>>>> "slave_ping_timeout": "15secs", >>>>> "slave_reregister_timeout": "10mins", >>>>> "user_sorter": "drf", >>>>> "version": "false", >>>>> "webui_dir": "/usr/share/mesos/webui", >>>>> "work_dir": "/var/lib/mesos", >>>>> "zk": "zk://localhost:2181/mesos", >>>>> "zk_session_timeout": "10secs" >>>>> }, >>>>> "frameworks": [ >>>>> { >>>>> "active": true, >>>>> "capabilities": [], >>>>> "checkpoint": false, >>>>> "completed_tasks": [], >>>>> "executors": [], >>>>> "failover_timeout": 0.0, >>>>> "hostname": "", >>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>>> "name": "GoDocker HTTP Framework", >>>>> "offered_resources": { >>>>> "cpus": 0.0, >>>>> "disk": 0.0, >>>>> "mem": 0.0 >>>>> }, >>>>> "offers": [], >>>>> "registered_time": 1465298174.2483, >>>>> "resources": { >>>>> "cpus": 0.0, >>>>> "disk": 0.0, >>>>> "mem": 0.0 >>>>> }, >>>>> "role": "*", >>>>> "tasks": [], >>>>> "unregistered_time": 0.0, >>>>> "used_resources": { >>>>> "cpus": 0.0, >>>>> "disk": 0.0, >>>>> "mem": 0.0 >>>>> }, >>>>> "user": "godocker_http_test", >>>>> "webui_url": "" >>>>> } >>>>> ], >>>>> "git_sha": "555db235a34afbb9fb49940376cc33a66f1f85f0", >>>>> "git_tag": "0.28.1", >>>>> "hostname": "tifenn.irisa.fr", >>>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956", >>>>> "leader": "[email protected]:5050", >>>>> "log_dir": "/var/log/mesos", >>>>> "orphan_tasks": [], >>>>> "pid": "[email protected]:5050", >>>>> "slaves": [ >>>>> { >>>>> "active": true, >>>>> "attributes": { >>>>> "hostname": "127.0.0.1" >>>>> }, >>>>> "hostname": "tifenn.irisa.fr", >>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0", >>>>> "offered_resources": { >>>>> "cpus": 0.0, >>>>> "disk": 0.0, >>>>> "mem": 0.0 >>>>> }, >>>>> "pid": "slave(1)@127.0.1.1:5051", >>>>> "registered_time": 1465298164.37517, >>>>> "reregistered_time": 1465298164.37526, >>>>> "reserved_resources": {}, >>>>> "resources": { >>>>> "cpus": 4.0, >>>>> "disk": 459470.0, >>>>> "mem": 14898.0, >>>>> "ports": "[31000-32000]" >>>>> }, >>>>> "unreserved_resources": { >>>>> "cpus": 4.0, >>>>> "disk": 459470.0, >>>>> "mem": 14898.0, >>>>> "ports": "[31000-32000]" >>>>> }, >>>>> "used_resources": { >>>>> "cpus": 0.0, >>>>> "disk": 0.0, >>>>> "mem": 0.0 >>>>> }, >>>>> "version": "0.28.1" >>>>> } >>>>> ], >>>>> "start_time": 1465298159.26321, >>>>> "unregistered_frameworks": [], >>>>> "version": "0.28.1" >>>>> } >>>>> >>>>> >>>>>> Thanks, >>>>>> >>>>>> Guangya >>>>>> >>>>>> On Tue, Jun 7, 2016 at 6:04 PM, Olivier Sallou < >>> [email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> I am trying to switch from Python to HTTP API. I use mesos 0.28.1 >>>>>>> >>>>>>> I could create framework to register, receive offers etc... but I >>> have >>>>>>> an issue accepting offers. >>>>>>> >>>>>>> I send my ACCEPT message but I do not receive any UPDATE message, only >>>>>>> new offers and hearbeat messages. >>>>>>> >>>>>>> On mesos master logs I see: >>>>>>> >>>>>>> I0607 11:45:15.873184 14896 http.cpp:312] HTTP POST for >>>>>>> /master/api/v1/scheduler from 127.0.0.1:38298 with >>>>>>> User-Agent='python-requests/2.9.1' >>>>>>> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call >>> for >>>>>>> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave >>>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 >>>>>>> (tifenn.irisa.fr) for framework >>>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020 (GoDocker HTTP Framework) >>>>>>> >>>>>>> There is a "Processing ACCEPT" and no error, but my task is not ran on >>>>>>> mesos. >>>>>>> No error on slave either. >>>>>>> >>>>>>> Response code to my ACCEPT is 202 as expected. >>>>>>> >>>>>>> Here is my HTTP json message: >>>>>>> >>>>>>> {'type': 'ACCEPT', >>>>>>> 'framework_id': {'value': >>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, >>>>>>> 'accept': { >>>>>>> 'operations': [ >>>>>>> {'type': 'LAUNCH', >>>>>>> 'launch': {'container': { >>>>>>> 'docker': {'image': u'centos:latest', >>>>>>> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, >>>>>>> 'type': 1, >>>>>>> 'volumes': [ >>>>>>> {'host_path': u'/a/b', 'container_path': >>>>>>> u'/mnt/home', 'mode': 1}, >>>>>>> {'host_path': u'/a/b/c', 'container_path': >>>>>>> u'/mnt/go-docker', 'mode': 1}, >>>>>>> {'host_path': u'/b/c/d', 'container_path': >>>>>>> u'/mnt/god-data', 'mode': 2} >>>>>>> ] >>>>>>> }, >>>>>>> 'name': u'testr', >>>>>>> 'task_id': {'value': '128'}, >>>>>>> 'command': {'uris': [{'value': >>>>>>> u'/home/osallou/docker.tar.gz'}], 'value': >>>>> u'/mnt/go-docker/wrapper.sh'}, >>>>>>> 'slave_id': {'value': >>>>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, >>>>>>> 'resources': [ >>>>>>> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, >>>>>>> {'scalar': {'value': 2000}, 'type': 0, 'name': 'mem'} >>>>>>> ] >>>>>>> } # end launch >>>>>>> } # end operation >>>>>>> ], >>>>>>> 'offer_ids': [{'value': >>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> There could be an issue with my task definition, but as no error is >>>>>>> raised and I receive no UPDATE error message..... >>>>>>> >>>>>>> Any hint on how to debug this? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Olivier Sallou >>>>>>> IRISA / University of Rennes 1 >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>>>>> Tel: 02.99.84.71.95 >>>>>>> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>>>> >>>>>>> >>>>> -- >>>>> Olivier Sallou >>>>> IRISA / University of Rennes 1 >>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>>> Tel: 02.99.84.71.95 >>>>> >>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>> >>>>> >>> -- >>> Olivier Sallou >>> IRISA / University of Rennes 1 >>> Campus de Beaulieu, 35000 RENNES - FRANCE >>> Tel: 02.99.84.71.95 >>> >>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>> >>> >>> > > -- > Olivier Sallou > IRISA / University of Rennes 1 > Campus de Beaulieu, 35000 RENNES - FRANCE > Tel: 02.99.84.71.95 > > gpg key id: 4096R/326D8438 (keyring.debian.org <http://keyring.debian.org/>) > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
