On 06/07/2016 04:53 PM, Guangya Liu wrote: > So how many agent nodes are there in your cluster? If you continue > receiving offer but without getting UPDATE message, then it may be caused > by that your task definition and the framework continually decline offer. I have only one node (master/slave), for development. It worked fine with the python API. we see on master that it received the ACCEPT, and no DECLINE. However, as I receive no UPDATE, I suppose that mesos "drops" the ACCEPT (wrong task definition maybe), and sends new offers several seconds after I sent the ACCEPT. > > Can you please share your framework code here for the logic of "Event:: > OFFERS"? Code is available here:
https://bitbucket.org/osallou/go-docker/src/b1948063fb7f68fbc77f5de6b473d832a7dd36af/plugins/mesos.py?at=master&fileviewer=file-view-default in method run of MesosThread, line 613 Code is a little complex, as it is a port of existing code using mesos python lib. Code related to HTTP is in development, so there may be further errors, but registration is fine as well as offer messages. I have added locally a debug print to show any message received by mesos (in case I would have received an other message indicating an error), but I received no other than offer and heartbeats. If Mesos see the ACCEPT message as it appears in logs, that it should either reject it (with a different status code than 202) or send an UPDATE error message if there is an error with my task definition. Olivier > > Thanks, > > Guangya > > On Tue, Jun 7, 2016 at 8:29 PM, Olivier Sallou <[email protected]> > wrote: > >> >> On 06/07/2016 01:59 PM, Guangya Liu wrote: >>> I can see that your framework is now holding the offer, how did you >> launch >>> task? >> I execute an HTTP POST request in Python with json content-type: >> >> {'type': 'ACCEPT', >> 'framework_id': {'value': u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, >> 'accept': { >> 'operations': [ >> {'type': 'LAUNCH', >> 'launch': {'container': { >> 'docker': {'image': u'centos:latest', >> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, >> 'type': 1, >> 'volumes': [ >> {'host_path': u'/a/b', 'container_path': >> u'/mnt/home', 'mode': 1}, >> {'host_path': u'/a/b/c', 'container_path': >> u'/mnt/go-docker', 'mode': 1}, >> {'host_path': u'/b/c/d', 'container_path': >> u'/mnt/god-data', 'mode': 2} >> ] >> }, >> 'name': u'testr', >> 'task_id': {'value': '128'}, >> 'command': {'uris': [{'value': >> u'/home/osallou/docker.tar.gz'}], 'value': u'/mnt/go-docker/wrapper.sh'}, >> 'slave_id': {'value': >> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, >> 'resources': [ >> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, >> {'scalar': {'value': 2000}, 'type': 0, 'name': 'mem'} >> ] >> } # end launch >> } # end operation >> ], >> 'offer_ids': [{'value': u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] >> } >> } >> >> We can see that Mesos received the ACCEPT: >> >> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call for >> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave >> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 >> (tifenn.irisa.fr) for framework >> >> >> and I continue to receive new offers, so "connection" is OK. I should >> receive an UPDATE message even if there is an error, but I receive none >> (I track/log all messages received, whatever the type). >> >> Olivier >> >>> Perhaps you can take a look at >>> https://github.com/apache/mesos/blob/master/src/cli/execute.cpp#L311 >> which >>> is an example framework using HTTP API >>> >>> Thanks, >>> >>> Guangya >>> >>> On Tue, Jun 7, 2016 at 7:19 PM, Olivier Sallou <[email protected]> >>> wrote: >>> >>>> On 06/07/2016 12:25 PM, Guangya Liu wrote: >>>>> Olivier, >>>>> >>>>> For such case, seems there is sth wrong with your framework? can you >>>> please >>>>> run the following two commands and check the output? >>>> I don't think it is a framework issue, I receive offers, heartbeats >> etc... >>>> It is only at task creation step, when I have no rejection nor update >>>> message. >>>> >>>> It could be (certainly) an issue with the json task message I sent in >>>> the ACCEPT, but as there is no error, I have no way to understand what's >>>> wrong with it. >>>>> curl "http://<mesos-master>:5050/master/frameworks" 2>/dev/null|python >>>> -m >>>>> json.tool >>>> { >>>> "completed_frameworks": [], >>>> "frameworks": [ >>>> { >>>> "active": true, >>>> "capabilities": [], >>>> "checkpoint": false, >>>> "completed_tasks": [], >>>> "executors": [], >>>> "failover_timeout": 0.0, >>>> "hostname": "", >>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>> "name": "GoDocker HTTP Framework", >>>> "offered_resources": { >>>> "cpus": 4.0, >>>> "disk": 459470.0, >>>> "mem": 14898.0, >>>> "ports": "[31000-32000]" >>>> }, >>>> "offers": [ >>>> { >>>> "framework_id": >>>> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956-O0", >>>> "resources": { >>>> "cpus": 4.0, >>>> "disk": 459470.0, >>>> "mem": 14898.0, >>>> "ports": "[31000-32000]" >>>> }, >>>> "slave_id": >> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0" >>>> } >>>> ], >>>> "registered_time": 1465298174.2483, >>>> "resources": { >>>> "cpus": 4.0, >>>> "disk": 459470.0, >>>> "mem": 14898.0, >>>> "ports": "[31000-32000]" >>>> }, >>>> "role": "*", >>>> "tasks": [], >>>> "unregistered_time": 0.0, >>>> "used_resources": { >>>> "cpus": 0.0, >>>> "disk": 0.0, >>>> "mem": 0.0 >>>> }, >>>> "user": "godocker_http_test", >>>> "webui_url": "" >>>> } >>>> ], >>>> "unregistered_frameworks": [] >>>> } >>>> >>>> >>>>> curl "http://<mesos-master>:5050/master/state" 2>/dev/null|python -m >>>>> json.tool >>>> { >>>> "activated_slaves": 1.0, >>>> "build_date": "2016-04-14 15:44:54", >>>> "build_time": 1460648694.0, >>>> "build_user": "root", >>>> "completed_frameworks": [], >>>> "deactivated_slaves": 0.0, >>>> "elected_time": 1465298164.01165, >>>> "flags": { >>>> "allocation_interval": "1secs", >>>> "allocator": "HierarchicalDRF", >>>> "authenticate": "false", >>>> "authenticate_http": "false", >>>> "authenticate_slaves": "false", >>>> "authenticators": "crammd5", >>>> "authorizers": "local", >>>> "framework_sorter": "drf", >>>> "help": "false", >>>> "hostname_lookup": "true", >>>> "http_authenticators": "basic", >>>> "initialize_driver_logging": "true", >>>> "log_auto_initialize": "true", >>>> "log_dir": "/var/log/mesos", >>>> "logbufsecs": "0", >>>> "logging_level": "INFO", >>>> "max_completed_frameworks": "50", >>>> "max_completed_tasks_per_framework": "1000", >>>> "max_slave_ping_timeouts": "5", >>>> "port": "5050", >>>> "quiet": "false", >>>> "quorum": "1", >>>> "recovery_slave_removal_limit": "100%", >>>> "registry": "replicated_log", >>>> "registry_fetch_timeout": "1mins", >>>> "registry_store_timeout": "20secs", >>>> "registry_strict": "false", >>>> "root_submissions": "true", >>>> "slave_ping_timeout": "15secs", >>>> "slave_reregister_timeout": "10mins", >>>> "user_sorter": "drf", >>>> "version": "false", >>>> "webui_dir": "/usr/share/mesos/webui", >>>> "work_dir": "/var/lib/mesos", >>>> "zk": "zk://localhost:2181/mesos", >>>> "zk_session_timeout": "10secs" >>>> }, >>>> "frameworks": [ >>>> { >>>> "active": true, >>>> "capabilities": [], >>>> "checkpoint": false, >>>> "completed_tasks": [], >>>> "executors": [], >>>> "failover_timeout": 0.0, >>>> "hostname": "", >>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>> "name": "GoDocker HTTP Framework", >>>> "offered_resources": { >>>> "cpus": 0.0, >>>> "disk": 0.0, >>>> "mem": 0.0 >>>> }, >>>> "offers": [], >>>> "registered_time": 1465298174.2483, >>>> "resources": { >>>> "cpus": 0.0, >>>> "disk": 0.0, >>>> "mem": 0.0 >>>> }, >>>> "role": "*", >>>> "tasks": [], >>>> "unregistered_time": 0.0, >>>> "used_resources": { >>>> "cpus": 0.0, >>>> "disk": 0.0, >>>> "mem": 0.0 >>>> }, >>>> "user": "godocker_http_test", >>>> "webui_url": "" >>>> } >>>> ], >>>> "git_sha": "555db235a34afbb9fb49940376cc33a66f1f85f0", >>>> "git_tag": "0.28.1", >>>> "hostname": "tifenn.irisa.fr", >>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956", >>>> "leader": "[email protected]:5050", >>>> "log_dir": "/var/log/mesos", >>>> "orphan_tasks": [], >>>> "pid": "[email protected]:5050", >>>> "slaves": [ >>>> { >>>> "active": true, >>>> "attributes": { >>>> "hostname": "127.0.0.1" >>>> }, >>>> "hostname": "tifenn.irisa.fr", >>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0", >>>> "offered_resources": { >>>> "cpus": 0.0, >>>> "disk": 0.0, >>>> "mem": 0.0 >>>> }, >>>> "pid": "slave(1)@127.0.1.1:5051", >>>> "registered_time": 1465298164.37517, >>>> "reregistered_time": 1465298164.37526, >>>> "reserved_resources": {}, >>>> "resources": { >>>> "cpus": 4.0, >>>> "disk": 459470.0, >>>> "mem": 14898.0, >>>> "ports": "[31000-32000]" >>>> }, >>>> "unreserved_resources": { >>>> "cpus": 4.0, >>>> "disk": 459470.0, >>>> "mem": 14898.0, >>>> "ports": "[31000-32000]" >>>> }, >>>> "used_resources": { >>>> "cpus": 0.0, >>>> "disk": 0.0, >>>> "mem": 0.0 >>>> }, >>>> "version": "0.28.1" >>>> } >>>> ], >>>> "start_time": 1465298159.26321, >>>> "unregistered_frameworks": [], >>>> "version": "0.28.1" >>>> } >>>> >>>> >>>>> Thanks, >>>>> >>>>> Guangya >>>>> >>>>> On Tue, Jun 7, 2016 at 6:04 PM, Olivier Sallou < >> [email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> I am trying to switch from Python to HTTP API. I use mesos 0.28.1 >>>>>> >>>>>> I could create framework to register, receive offers etc... but I >> have >>>>>> an issue accepting offers. >>>>>> >>>>>> I send my ACCEPT message but I do not receive any UPDATE message, only >>>>>> new offers and hearbeat messages. >>>>>> >>>>>> On mesos master logs I see: >>>>>> >>>>>> I0607 11:45:15.873184 14896 http.cpp:312] HTTP POST for >>>>>> /master/api/v1/scheduler from 127.0.0.1:38298 with >>>>>> User-Agent='python-requests/2.9.1' >>>>>> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call >> for >>>>>> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave >>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 >>>>>> (tifenn.irisa.fr) for framework >>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020 (GoDocker HTTP Framework) >>>>>> >>>>>> There is a "Processing ACCEPT" and no error, but my task is not ran on >>>>>> mesos. >>>>>> No error on slave either. >>>>>> >>>>>> Response code to my ACCEPT is 202 as expected. >>>>>> >>>>>> Here is my HTTP json message: >>>>>> >>>>>> {'type': 'ACCEPT', >>>>>> 'framework_id': {'value': >> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, >>>>>> 'accept': { >>>>>> 'operations': [ >>>>>> {'type': 'LAUNCH', >>>>>> 'launch': {'container': { >>>>>> 'docker': {'image': u'centos:latest', >>>>>> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, >>>>>> 'type': 1, >>>>>> 'volumes': [ >>>>>> {'host_path': u'/a/b', 'container_path': >>>>>> u'/mnt/home', 'mode': 1}, >>>>>> {'host_path': u'/a/b/c', 'container_path': >>>>>> u'/mnt/go-docker', 'mode': 1}, >>>>>> {'host_path': u'/b/c/d', 'container_path': >>>>>> u'/mnt/god-data', 'mode': 2} >>>>>> ] >>>>>> }, >>>>>> 'name': u'testr', >>>>>> 'task_id': {'value': '128'}, >>>>>> 'command': {'uris': [{'value': >>>>>> u'/home/osallou/docker.tar.gz'}], 'value': >>>> u'/mnt/go-docker/wrapper.sh'}, >>>>>> 'slave_id': {'value': >>>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, >>>>>> 'resources': [ >>>>>> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, >>>>>> {'scalar': {'value': 2000}, 'type': 0, 'name': 'mem'} >>>>>> ] >>>>>> } # end launch >>>>>> } # end operation >>>>>> ], >>>>>> 'offer_ids': [{'value': >>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] >>>>>> } >>>>>> } >>>>>> >>>>>> There could be an issue with my task definition, but as no error is >>>>>> raised and I receive no UPDATE error message..... >>>>>> >>>>>> Any hint on how to debug this? >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> -- >>>>>> Olivier Sallou >>>>>> IRISA / University of Rennes 1 >>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>>>> Tel: 02.99.84.71.95 >>>>>> >>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>>> >>>>>> >>>> -- >>>> Olivier Sallou >>>> IRISA / University of Rennes 1 >>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>> Tel: 02.99.84.71.95 >>>> >>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>> >>>> >> -- >> Olivier Sallou >> IRISA / University of Rennes 1 >> Campus de Beaulieu, 35000 RENNES - FRANCE >> Tel: 02.99.84.71.95 >> >> gpg key id: 4096R/326D8438 (keyring.debian.org) >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >> >> >> -- Olivier Sallou IRISA / University of Rennes 1 Campus de Beaulieu, 35000 RENNES - FRANCE Tel: 02.99.84.71.95 gpg key id: 4096R/326D8438 (keyring.debian.org) Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
