On 06/07/2016 05:46 PM, Anand Mazumdar wrote: > Olivier, > > You are missing the “task_infos” key in your “ACCEPT” call. The master treats > “Accept” operations with no launch tasks as declining offers implicitly. I > would file a followup JIRA to ensure this is logged on the master (if not so). > > An example correct JSON: > https://gist.github.com/hatred/7325d8a4afde607ecc0f376ab62d60eb > <https://gist.github.com/hatred/7325d8a4afde607ecc0f376ab62d60eb> thanks, example is really useful.
I supposed task "structure" was the issue, but getting no error log on master about this was an issue and difficult to understand. Would indeed be fine to get a master log about the issue. Thanks! Olivier > > -anand > >> On Jun 7, 2016, at 8:38 AM, Olivier Sallou <[email protected]> wrote: >> >> >> >> On 06/07/2016 04:53 PM, Guangya Liu wrote: >>> So how many agent nodes are there in your cluster? If you continue >>> receiving offer but without getting UPDATE message, then it may be caused >>> by that your task definition and the framework continually decline offer. >> I have only one node (master/slave), for development. It worked fine >> with the python API. >> we see on master that it received the ACCEPT, and no DECLINE. However, >> as I receive no UPDATE, I suppose that mesos "drops" the ACCEPT (wrong >> task definition maybe), and sends new offers several seconds after I >> sent the ACCEPT. >>> Can you please share your framework code here for the logic of "Event:: >>> OFFERS"? >> Code is available here: >> >> https://bitbucket.org/osallou/go-docker/src/b1948063fb7f68fbc77f5de6b473d832a7dd36af/plugins/mesos.py?at=master&fileviewer=file-view-default >> >> <https://bitbucket.org/osallou/go-docker/src/b1948063fb7f68fbc77f5de6b473d832a7dd36af/plugins/mesos.py?at=master&fileviewer=file-view-default> >> >> in method run of MesosThread, line 613 >> >> Code is a little complex, as it is a port of existing code using mesos >> python lib. >> >> Code related to HTTP is in development, so there may be further errors, >> but registration is fine as well as offer messages. >> >> I have added locally a debug print to show any message received by mesos >> (in case I would have received an other message indicating an error), >> but I received no other than offer and heartbeats. >> >> If Mesos see the ACCEPT message as it appears in logs, that it should >> either reject it (with a different status code than 202) or send an >> UPDATE error message if there is an error with my task definition. >> >> Olivier >>> Thanks, >>> >>> Guangya >>> >>> On Tue, Jun 7, 2016 at 8:29 PM, Olivier Sallou <[email protected]> >>> wrote: >>> >>>> On 06/07/2016 01:59 PM, Guangya Liu wrote: >>>>> I can see that your framework is now holding the offer, how did you >>>> launch >>>>> task? >>>> I execute an HTTP POST request in Python with json content-type: >>>> >>>> {'type': 'ACCEPT', >>>> 'framework_id': {'value': u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, >>>> 'accept': { >>>> 'operations': [ >>>> {'type': 'LAUNCH', >>>> 'launch': {'container': { >>>> 'docker': {'image': u'centos:latest', >>>> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, >>>> 'type': 1, >>>> 'volumes': [ >>>> {'host_path': u'/a/b', 'container_path': >>>> u'/mnt/home', 'mode': 1}, >>>> {'host_path': u'/a/b/c', 'container_path': >>>> u'/mnt/go-docker', 'mode': 1}, >>>> {'host_path': u'/b/c/d', 'container_path': >>>> u'/mnt/god-data', 'mode': 2} >>>> ] >>>> }, >>>> 'name': u'testr', >>>> 'task_id': {'value': '128'}, >>>> 'command': {'uris': [{'value': >>>> u'/home/osallou/docker.tar.gz'}], 'value': u'/mnt/go-docker/wrapper.sh'}, >>>> 'slave_id': {'value': >>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, >>>> 'resources': [ >>>> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, >>>> {'scalar': {'value': 2000}, 'type': 0, 'name': 'mem'} >>>> ] >>>> } # end launch >>>> } # end operation >>>> ], >>>> 'offer_ids': [{'value': u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] >>>> } >>>> } >>>> >>>> We can see that Mesos received the ACCEPT: >>>> >>>> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call for >>>> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave >>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 >>>> (tifenn.irisa.fr) for framework >>>> >>>> >>>> and I continue to receive new offers, so "connection" is OK. I should >>>> receive an UPDATE message even if there is an error, but I receive none >>>> (I track/log all messages received, whatever the type). >>>> >>>> Olivier >>>> >>>>> Perhaps you can take a look at >>>>> https://github.com/apache/mesos/blob/master/src/cli/execute.cpp#L311 >>>> which >>>>> is an example framework using HTTP API >>>>> >>>>> Thanks, >>>>> >>>>> Guangya >>>>> >>>>> On Tue, Jun 7, 2016 at 7:19 PM, Olivier Sallou <[email protected]> >>>>> wrote: >>>>> >>>>>> On 06/07/2016 12:25 PM, Guangya Liu wrote: >>>>>>> Olivier, >>>>>>> >>>>>>> For such case, seems there is sth wrong with your framework? can you >>>>>> please >>>>>>> run the following two commands and check the output? >>>>>> I don't think it is a framework issue, I receive offers, heartbeats >>>> etc... >>>>>> It is only at task creation step, when I have no rejection nor update >>>>>> message. >>>>>> >>>>>> It could be (certainly) an issue with the json task message I sent in >>>>>> the ACCEPT, but as there is no error, I have no way to understand what's >>>>>> wrong with it. >>>>>>> curl "http://<mesos-master>:5050/master/frameworks" 2>/dev/null|python >>>>>> -m >>>>>>> json.tool >>>>>> { >>>>>> "completed_frameworks": [], >>>>>> "frameworks": [ >>>>>> { >>>>>> "active": true, >>>>>> "capabilities": [], >>>>>> "checkpoint": false, >>>>>> "completed_tasks": [], >>>>>> "executors": [], >>>>>> "failover_timeout": 0.0, >>>>>> "hostname": "", >>>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>>>> "name": "GoDocker HTTP Framework", >>>>>> "offered_resources": { >>>>>> "cpus": 4.0, >>>>>> "disk": 459470.0, >>>>>> "mem": 14898.0, >>>>>> "ports": "[31000-32000]" >>>>>> }, >>>>>> "offers": [ >>>>>> { >>>>>> "framework_id": >>>>>> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956-O0", >>>>>> "resources": { >>>>>> "cpus": 4.0, >>>>>> "disk": 459470.0, >>>>>> "mem": 14898.0, >>>>>> "ports": "[31000-32000]" >>>>>> }, >>>>>> "slave_id": >>>> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0" >>>>>> } >>>>>> ], >>>>>> "registered_time": 1465298174.2483, >>>>>> "resources": { >>>>>> "cpus": 4.0, >>>>>> "disk": 459470.0, >>>>>> "mem": 14898.0, >>>>>> "ports": "[31000-32000]" >>>>>> }, >>>>>> "role": "*", >>>>>> "tasks": [], >>>>>> "unregistered_time": 0.0, >>>>>> "used_resources": { >>>>>> "cpus": 0.0, >>>>>> "disk": 0.0, >>>>>> "mem": 0.0 >>>>>> }, >>>>>> "user": "godocker_http_test", >>>>>> "webui_url": "" >>>>>> } >>>>>> ], >>>>>> "unregistered_frameworks": [] >>>>>> } >>>>>> >>>>>> >>>>>>> curl "http://<mesos-master>:5050/master/state" 2>/dev/null|python -m >>>>>>> json.tool >>>>>> { >>>>>> "activated_slaves": 1.0, >>>>>> "build_date": "2016-04-14 15:44:54", >>>>>> "build_time": 1460648694.0, >>>>>> "build_user": "root", >>>>>> "completed_frameworks": [], >>>>>> "deactivated_slaves": 0.0, >>>>>> "elected_time": 1465298164.01165, >>>>>> "flags": { >>>>>> "allocation_interval": "1secs", >>>>>> "allocator": "HierarchicalDRF", >>>>>> "authenticate": "false", >>>>>> "authenticate_http": "false", >>>>>> "authenticate_slaves": "false", >>>>>> "authenticators": "crammd5", >>>>>> "authorizers": "local", >>>>>> "framework_sorter": "drf", >>>>>> "help": "false", >>>>>> "hostname_lookup": "true", >>>>>> "http_authenticators": "basic", >>>>>> "initialize_driver_logging": "true", >>>>>> "log_auto_initialize": "true", >>>>>> "log_dir": "/var/log/mesos", >>>>>> "logbufsecs": "0", >>>>>> "logging_level": "INFO", >>>>>> "max_completed_frameworks": "50", >>>>>> "max_completed_tasks_per_framework": "1000", >>>>>> "max_slave_ping_timeouts": "5", >>>>>> "port": "5050", >>>>>> "quiet": "false", >>>>>> "quorum": "1", >>>>>> "recovery_slave_removal_limit": "100%", >>>>>> "registry": "replicated_log", >>>>>> "registry_fetch_timeout": "1mins", >>>>>> "registry_store_timeout": "20secs", >>>>>> "registry_strict": "false", >>>>>> "root_submissions": "true", >>>>>> "slave_ping_timeout": "15secs", >>>>>> "slave_reregister_timeout": "10mins", >>>>>> "user_sorter": "drf", >>>>>> "version": "false", >>>>>> "webui_dir": "/usr/share/mesos/webui", >>>>>> "work_dir": "/var/lib/mesos", >>>>>> "zk": "zk://localhost:2181/mesos", >>>>>> "zk_session_timeout": "10secs" >>>>>> }, >>>>>> "frameworks": [ >>>>>> { >>>>>> "active": true, >>>>>> "capabilities": [], >>>>>> "checkpoint": false, >>>>>> "completed_tasks": [], >>>>>> "executors": [], >>>>>> "failover_timeout": 0.0, >>>>>> "hostname": "", >>>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>>>> "name": "GoDocker HTTP Framework", >>>>>> "offered_resources": { >>>>>> "cpus": 0.0, >>>>>> "disk": 0.0, >>>>>> "mem": 0.0 >>>>>> }, >>>>>> "offers": [], >>>>>> "registered_time": 1465298174.2483, >>>>>> "resources": { >>>>>> "cpus": 0.0, >>>>>> "disk": 0.0, >>>>>> "mem": 0.0 >>>>>> }, >>>>>> "role": "*", >>>>>> "tasks": [], >>>>>> "unregistered_time": 0.0, >>>>>> "used_resources": { >>>>>> "cpus": 0.0, >>>>>> "disk": 0.0, >>>>>> "mem": 0.0 >>>>>> }, >>>>>> "user": "godocker_http_test", >>>>>> "webui_url": "" >>>>>> } >>>>>> ], >>>>>> "git_sha": "555db235a34afbb9fb49940376cc33a66f1f85f0", >>>>>> "git_tag": "0.28.1", >>>>>> "hostname": "tifenn.irisa.fr", >>>>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956", >>>>>> "leader": "[email protected]:5050", >>>>>> "log_dir": "/var/log/mesos", >>>>>> "orphan_tasks": [], >>>>>> "pid": "[email protected]:5050", >>>>>> "slaves": [ >>>>>> { >>>>>> "active": true, >>>>>> "attributes": { >>>>>> "hostname": "127.0.0.1" >>>>>> }, >>>>>> "hostname": "tifenn.irisa.fr", >>>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0", >>>>>> "offered_resources": { >>>>>> "cpus": 0.0, >>>>>> "disk": 0.0, >>>>>> "mem": 0.0 >>>>>> }, >>>>>> "pid": "slave(1)@127.0.1.1:5051", >>>>>> "registered_time": 1465298164.37517, >>>>>> "reregistered_time": 1465298164.37526, >>>>>> "reserved_resources": {}, >>>>>> "resources": { >>>>>> "cpus": 4.0, >>>>>> "disk": 459470.0, >>>>>> "mem": 14898.0, >>>>>> "ports": "[31000-32000]" >>>>>> }, >>>>>> "unreserved_resources": { >>>>>> "cpus": 4.0, >>>>>> "disk": 459470.0, >>>>>> "mem": 14898.0, >>>>>> "ports": "[31000-32000]" >>>>>> }, >>>>>> "used_resources": { >>>>>> "cpus": 0.0, >>>>>> "disk": 0.0, >>>>>> "mem": 0.0 >>>>>> }, >>>>>> "version": "0.28.1" >>>>>> } >>>>>> ], >>>>>> "start_time": 1465298159.26321, >>>>>> "unregistered_frameworks": [], >>>>>> "version": "0.28.1" >>>>>> } >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Guangya >>>>>>> >>>>>>> On Tue, Jun 7, 2016 at 6:04 PM, Olivier Sallou < >>>> [email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> I am trying to switch from Python to HTTP API. I use mesos 0.28.1 >>>>>>>> >>>>>>>> I could create framework to register, receive offers etc... but I >>>> have >>>>>>>> an issue accepting offers. >>>>>>>> >>>>>>>> I send my ACCEPT message but I do not receive any UPDATE message, only >>>>>>>> new offers and hearbeat messages. >>>>>>>> >>>>>>>> On mesos master logs I see: >>>>>>>> >>>>>>>> I0607 11:45:15.873184 14896 http.cpp:312] HTTP POST for >>>>>>>> /master/api/v1/scheduler from 127.0.0.1:38298 with >>>>>>>> User-Agent='python-requests/2.9.1' >>>>>>>> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call >>>> for >>>>>>>> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave >>>>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 >>>>>>>> (tifenn.irisa.fr) for framework >>>>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020 (GoDocker HTTP Framework) >>>>>>>> >>>>>>>> There is a "Processing ACCEPT" and no error, but my task is not ran on >>>>>>>> mesos. >>>>>>>> No error on slave either. >>>>>>>> >>>>>>>> Response code to my ACCEPT is 202 as expected. >>>>>>>> >>>>>>>> Here is my HTTP json message: >>>>>>>> >>>>>>>> {'type': 'ACCEPT', >>>>>>>> 'framework_id': {'value': >>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, >>>>>>>> 'accept': { >>>>>>>> 'operations': [ >>>>>>>> {'type': 'LAUNCH', >>>>>>>> 'launch': {'container': { >>>>>>>> 'docker': {'image': u'centos:latest', >>>>>>>> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, >>>>>>>> 'type': 1, >>>>>>>> 'volumes': [ >>>>>>>> {'host_path': u'/a/b', 'container_path': >>>>>>>> u'/mnt/home', 'mode': 1}, >>>>>>>> {'host_path': u'/a/b/c', 'container_path': >>>>>>>> u'/mnt/go-docker', 'mode': 1}, >>>>>>>> {'host_path': u'/b/c/d', 'container_path': >>>>>>>> u'/mnt/god-data', 'mode': 2} >>>>>>>> ] >>>>>>>> }, >>>>>>>> 'name': u'testr', >>>>>>>> 'task_id': {'value': '128'}, >>>>>>>> 'command': {'uris': [{'value': >>>>>>>> u'/home/osallou/docker.tar.gz'}], 'value': >>>>>> u'/mnt/go-docker/wrapper.sh'}, >>>>>>>> 'slave_id': {'value': >>>>>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, >>>>>>>> 'resources': [ >>>>>>>> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, >>>>>>>> {'scalar': {'value': 2000}, 'type': 0, 'name': 'mem'} >>>>>>>> ] >>>>>>>> } # end launch >>>>>>>> } # end operation >>>>>>>> ], >>>>>>>> 'offer_ids': [{'value': >>>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> There could be an issue with my task definition, but as no error is >>>>>>>> raised and I receive no UPDATE error message..... >>>>>>>> >>>>>>>> Any hint on how to debug this? >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Olivier Sallou >>>>>>>> IRISA / University of Rennes 1 >>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>>>>>> Tel: 02.99.84.71.95 >>>>>>>> >>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>>>>> >>>>>>>> >>>>>> -- >>>>>> Olivier Sallou >>>>>> IRISA / University of Rennes 1 >>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>>>> Tel: 02.99.84.71.95 >>>>>> >>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>>> >>>>>> >>>> -- >>>> Olivier Sallou >>>> IRISA / University of Rennes 1 >>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>> Tel: 02.99.84.71.95 >>>> >>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>> >>>> >>>> >> -- >> Olivier Sallou >> IRISA / University of Rennes 1 >> Campus de Beaulieu, 35000 RENNES - FRANCE >> Tel: 02.99.84.71.95 >> >> gpg key id: 4096R/326D8438 (keyring.debian.org <http://keyring.debian.org/>) >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > -- Olivier Sallou IRISA / University of Rennes 1 Campus de Beaulieu, 35000 RENNES - FRANCE Tel: 02.99.84.71.95 gpg key id: 4096R/326D8438 (keyring.debian.org) Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
