On 06/07/2016 06:29 PM, Vinod Kone wrote: > Olivier, on a side note, it's great to see that you are playing with the > new HTTP API in python! I briefly looked at your linked code and it looks > like you are mixing the business logic of your application and the Mesos > API interaction in the same file. It would be great if (at some point) you > can extract the Mesos API interaction into a python library that can be > used by other frameworks. See other libraries (C++ > <https://github.com/apache/mesos/blob/master/include/mesos/v1/scheduler.hpp>, > Java <https://github.com/mesosphere/mesos-rxjava>, Go > <https://github.com/mesos/mesos-go>) for inspiration. I will try to do this later on. For the moment I focus no reproducing my code with the HTTP API, code is not yet clean. Would be better indeed to extract Mesos side from business logic.
Olivier > > On Tue, Jun 7, 2016 at 11:46 AM, Anand Mazumdar <an...@mesosphere.io> wrote: > >> Olivier, >> >> You are missing the “task_infos” key in your “ACCEPT” call. The master >> treats “Accept” operations with no launch tasks as declining offers >> implicitly. I would file a followup JIRA to ensure this is logged on the >> master (if not so). >> >> An example correct JSON: >> https://gist.github.com/hatred/7325d8a4afde607ecc0f376ab62d60eb < >> https://gist.github.com/hatred/7325d8a4afde607ecc0f376ab62d60eb> >> >> -anand >> >>> On Jun 7, 2016, at 8:38 AM, Olivier Sallou <olivier.sal...@irisa.fr> >> wrote: >>> >>> >>> On 06/07/2016 04:53 PM, Guangya Liu wrote: >>>> So how many agent nodes are there in your cluster? If you continue >>>> receiving offer but without getting UPDATE message, then it may be >> caused >>>> by that your task definition and the framework continually decline >> offer. >>> I have only one node (master/slave), for development. It worked fine >>> with the python API. >>> we see on master that it received the ACCEPT, and no DECLINE. However, >>> as I receive no UPDATE, I suppose that mesos "drops" the ACCEPT (wrong >>> task definition maybe), and sends new offers several seconds after I >>> sent the ACCEPT. >>>> Can you please share your framework code here for the logic of "Event:: >>>> OFFERS"? >>> Code is available here: >>> >>> >> https://bitbucket.org/osallou/go-docker/src/b1948063fb7f68fbc77f5de6b473d832a7dd36af/plugins/mesos.py?at=master&fileviewer=file-view-default >> < >> https://bitbucket.org/osallou/go-docker/src/b1948063fb7f68fbc77f5de6b473d832a7dd36af/plugins/mesos.py?at=master&fileviewer=file-view-default >>> >>> in method run of MesosThread, line 613 >>> >>> Code is a little complex, as it is a port of existing code using mesos >>> python lib. >>> >>> Code related to HTTP is in development, so there may be further errors, >>> but registration is fine as well as offer messages. >>> >>> I have added locally a debug print to show any message received by mesos >>> (in case I would have received an other message indicating an error), >>> but I received no other than offer and heartbeats. >>> >>> If Mesos see the ACCEPT message as it appears in logs, that it should >>> either reject it (with a different status code than 202) or send an >>> UPDATE error message if there is an error with my task definition. >>> >>> Olivier >>>> Thanks, >>>> >>>> Guangya >>>> >>>> On Tue, Jun 7, 2016 at 8:29 PM, Olivier Sallou <olivier.sal...@irisa.fr >>>> wrote: >>>> >>>>> On 06/07/2016 01:59 PM, Guangya Liu wrote: >>>>>> I can see that your framework is now holding the offer, how did you >>>>> launch >>>>>> task? >>>>> I execute an HTTP POST request in Python with json content-type: >>>>> >>>>> {'type': 'ACCEPT', >>>>> 'framework_id': {'value': >> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, >>>>> 'accept': { >>>>> 'operations': [ >>>>> {'type': 'LAUNCH', >>>>> 'launch': {'container': { >>>>> 'docker': {'image': u'centos:latest', >>>>> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, >>>>> 'type': 1, >>>>> 'volumes': [ >>>>> {'host_path': u'/a/b', 'container_path': >>>>> u'/mnt/home', 'mode': 1}, >>>>> {'host_path': u'/a/b/c', 'container_path': >>>>> u'/mnt/go-docker', 'mode': 1}, >>>>> {'host_path': u'/b/c/d', 'container_path': >>>>> u'/mnt/god-data', 'mode': 2} >>>>> ] >>>>> }, >>>>> 'name': u'testr', >>>>> 'task_id': {'value': '128'}, >>>>> 'command': {'uris': [{'value': >>>>> u'/home/osallou/docker.tar.gz'}], 'value': >> u'/mnt/go-docker/wrapper.sh'}, >>>>> 'slave_id': {'value': >>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, >>>>> 'resources': [ >>>>> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, >>>>> {'scalar': {'value': 2000}, 'type': 0, 'name': 'mem'} >>>>> ] >>>>> } # end launch >>>>> } # end operation >>>>> ], >>>>> 'offer_ids': [{'value': >> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] >>>>> } >>>>> } >>>>> >>>>> We can see that Mesos received the ACCEPT: >>>>> >>>>> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call for >>>>> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave >>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 >>>>> (tifenn.irisa.fr) for framework >>>>> >>>>> >>>>> and I continue to receive new offers, so "connection" is OK. I should >>>>> receive an UPDATE message even if there is an error, but I receive none >>>>> (I track/log all messages received, whatever the type). >>>>> >>>>> Olivier >>>>> >>>>>> Perhaps you can take a look at >>>>>> https://github.com/apache/mesos/blob/master/src/cli/execute.cpp#L311 >>>>> which >>>>>> is an example framework using HTTP API >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Guangya >>>>>> >>>>>> On Tue, Jun 7, 2016 at 7:19 PM, Olivier Sallou < >> olivier.sal...@irisa.fr> >>>>>> wrote: >>>>>> >>>>>>> On 06/07/2016 12:25 PM, Guangya Liu wrote: >>>>>>>> Olivier, >>>>>>>> >>>>>>>> For such case, seems there is sth wrong with your framework? can you >>>>>>> please >>>>>>>> run the following two commands and check the output? >>>>>>> I don't think it is a framework issue, I receive offers, heartbeats >>>>> etc... >>>>>>> It is only at task creation step, when I have no rejection nor update >>>>>>> message. >>>>>>> >>>>>>> It could be (certainly) an issue with the json task message I sent in >>>>>>> the ACCEPT, but as there is no error, I have no way to understand >> what's >>>>>>> wrong with it. >>>>>>>> curl "http://<mesos-master>:5050/master/frameworks" >> 2>/dev/null|python >>>>>>> -m >>>>>>>> json.tool >>>>>>> { >>>>>>> "completed_frameworks": [], >>>>>>> "frameworks": [ >>>>>>> { >>>>>>> "active": true, >>>>>>> "capabilities": [], >>>>>>> "checkpoint": false, >>>>>>> "completed_tasks": [], >>>>>>> "executors": [], >>>>>>> "failover_timeout": 0.0, >>>>>>> "hostname": "", >>>>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>>>>> "name": "GoDocker HTTP Framework", >>>>>>> "offered_resources": { >>>>>>> "cpus": 4.0, >>>>>>> "disk": 459470.0, >>>>>>> "mem": 14898.0, >>>>>>> "ports": "[31000-32000]" >>>>>>> }, >>>>>>> "offers": [ >>>>>>> { >>>>>>> "framework_id": >>>>>>> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>>>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956-O0", >>>>>>> "resources": { >>>>>>> "cpus": 4.0, >>>>>>> "disk": 459470.0, >>>>>>> "mem": 14898.0, >>>>>>> "ports": "[31000-32000]" >>>>>>> }, >>>>>>> "slave_id": >>>>> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0" >>>>>>> } >>>>>>> ], >>>>>>> "registered_time": 1465298174.2483, >>>>>>> "resources": { >>>>>>> "cpus": 4.0, >>>>>>> "disk": 459470.0, >>>>>>> "mem": 14898.0, >>>>>>> "ports": "[31000-32000]" >>>>>>> }, >>>>>>> "role": "*", >>>>>>> "tasks": [], >>>>>>> "unregistered_time": 0.0, >>>>>>> "used_resources": { >>>>>>> "cpus": 0.0, >>>>>>> "disk": 0.0, >>>>>>> "mem": 0.0 >>>>>>> }, >>>>>>> "user": "godocker_http_test", >>>>>>> "webui_url": "" >>>>>>> } >>>>>>> ], >>>>>>> "unregistered_frameworks": [] >>>>>>> } >>>>>>> >>>>>>> >>>>>>>> curl "http://<mesos-master>:5050/master/state" 2>/dev/null|python >> -m >>>>>>>> json.tool >>>>>>> { >>>>>>> "activated_slaves": 1.0, >>>>>>> "build_date": "2016-04-14 15:44:54", >>>>>>> "build_time": 1460648694.0, >>>>>>> "build_user": "root", >>>>>>> "completed_frameworks": [], >>>>>>> "deactivated_slaves": 0.0, >>>>>>> "elected_time": 1465298164.01165, >>>>>>> "flags": { >>>>>>> "allocation_interval": "1secs", >>>>>>> "allocator": "HierarchicalDRF", >>>>>>> "authenticate": "false", >>>>>>> "authenticate_http": "false", >>>>>>> "authenticate_slaves": "false", >>>>>>> "authenticators": "crammd5", >>>>>>> "authorizers": "local", >>>>>>> "framework_sorter": "drf", >>>>>>> "help": "false", >>>>>>> "hostname_lookup": "true", >>>>>>> "http_authenticators": "basic", >>>>>>> "initialize_driver_logging": "true", >>>>>>> "log_auto_initialize": "true", >>>>>>> "log_dir": "/var/log/mesos", >>>>>>> "logbufsecs": "0", >>>>>>> "logging_level": "INFO", >>>>>>> "max_completed_frameworks": "50", >>>>>>> "max_completed_tasks_per_framework": "1000", >>>>>>> "max_slave_ping_timeouts": "5", >>>>>>> "port": "5050", >>>>>>> "quiet": "false", >>>>>>> "quorum": "1", >>>>>>> "recovery_slave_removal_limit": "100%", >>>>>>> "registry": "replicated_log", >>>>>>> "registry_fetch_timeout": "1mins", >>>>>>> "registry_store_timeout": "20secs", >>>>>>> "registry_strict": "false", >>>>>>> "root_submissions": "true", >>>>>>> "slave_ping_timeout": "15secs", >>>>>>> "slave_reregister_timeout": "10mins", >>>>>>> "user_sorter": "drf", >>>>>>> "version": "false", >>>>>>> "webui_dir": "/usr/share/mesos/webui", >>>>>>> "work_dir": "/var/lib/mesos", >>>>>>> "zk": "zk://localhost:2181/mesos", >>>>>>> "zk_session_timeout": "10secs" >>>>>>> }, >>>>>>> "frameworks": [ >>>>>>> { >>>>>>> "active": true, >>>>>>> "capabilities": [], >>>>>>> "checkpoint": false, >>>>>>> "completed_tasks": [], >>>>>>> "executors": [], >>>>>>> "failover_timeout": 0.0, >>>>>>> "hostname": "", >>>>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", >>>>>>> "name": "GoDocker HTTP Framework", >>>>>>> "offered_resources": { >>>>>>> "cpus": 0.0, >>>>>>> "disk": 0.0, >>>>>>> "mem": 0.0 >>>>>>> }, >>>>>>> "offers": [], >>>>>>> "registered_time": 1465298174.2483, >>>>>>> "resources": { >>>>>>> "cpus": 0.0, >>>>>>> "disk": 0.0, >>>>>>> "mem": 0.0 >>>>>>> }, >>>>>>> "role": "*", >>>>>>> "tasks": [], >>>>>>> "unregistered_time": 0.0, >>>>>>> "used_resources": { >>>>>>> "cpus": 0.0, >>>>>>> "disk": 0.0, >>>>>>> "mem": 0.0 >>>>>>> }, >>>>>>> "user": "godocker_http_test", >>>>>>> "webui_url": "" >>>>>>> } >>>>>>> ], >>>>>>> "git_sha": "555db235a34afbb9fb49940376cc33a66f1f85f0", >>>>>>> "git_tag": "0.28.1", >>>>>>> "hostname": "tifenn.irisa.fr", >>>>>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956", >>>>>>> "leader": "master@127.0.1.1:5050", >>>>>>> "log_dir": "/var/log/mesos", >>>>>>> "orphan_tasks": [], >>>>>>> "pid": "master@127.0.1.1:5050", >>>>>>> "slaves": [ >>>>>>> { >>>>>>> "active": true, >>>>>>> "attributes": { >>>>>>> "hostname": "127.0.0.1" >>>>>>> }, >>>>>>> "hostname": "tifenn.irisa.fr", >>>>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0", >>>>>>> "offered_resources": { >>>>>>> "cpus": 0.0, >>>>>>> "disk": 0.0, >>>>>>> "mem": 0.0 >>>>>>> }, >>>>>>> "pid": "slave(1)@127.0.1.1:5051", >>>>>>> "registered_time": 1465298164.37517, >>>>>>> "reregistered_time": 1465298164.37526, >>>>>>> "reserved_resources": {}, >>>>>>> "resources": { >>>>>>> "cpus": 4.0, >>>>>>> "disk": 459470.0, >>>>>>> "mem": 14898.0, >>>>>>> "ports": "[31000-32000]" >>>>>>> }, >>>>>>> "unreserved_resources": { >>>>>>> "cpus": 4.0, >>>>>>> "disk": 459470.0, >>>>>>> "mem": 14898.0, >>>>>>> "ports": "[31000-32000]" >>>>>>> }, >>>>>>> "used_resources": { >>>>>>> "cpus": 0.0, >>>>>>> "disk": 0.0, >>>>>>> "mem": 0.0 >>>>>>> }, >>>>>>> "version": "0.28.1" >>>>>>> } >>>>>>> ], >>>>>>> "start_time": 1465298159.26321, >>>>>>> "unregistered_frameworks": [], >>>>>>> "version": "0.28.1" >>>>>>> } >>>>>>> >>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Guangya >>>>>>>> >>>>>>>> On Tue, Jun 7, 2016 at 6:04 PM, Olivier Sallou < >>>>> olivier.sal...@irisa.fr> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> I am trying to switch from Python to HTTP API. I use mesos 0.28.1 >>>>>>>>> >>>>>>>>> I could create framework to register, receive offers etc... but I >>>>> have >>>>>>>>> an issue accepting offers. >>>>>>>>> >>>>>>>>> I send my ACCEPT message but I do not receive any UPDATE message, >> only >>>>>>>>> new offers and hearbeat messages. >>>>>>>>> >>>>>>>>> On mesos master logs I see: >>>>>>>>> >>>>>>>>> I0607 11:45:15.873184 14896 http.cpp:312] HTTP POST for >>>>>>>>> /master/api/v1/scheduler from 127.0.0.1:38298 with >>>>>>>>> User-Agent='python-requests/2.9.1' >>>>>>>>> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call >>>>> for >>>>>>>>> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave >>>>>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 >>>>>>>>> (tifenn.irisa.fr) for framework >>>>>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020 (GoDocker HTTP Framework) >>>>>>>>> >>>>>>>>> There is a "Processing ACCEPT" and no error, but my task is not >> ran on >>>>>>>>> mesos. >>>>>>>>> No error on slave either. >>>>>>>>> >>>>>>>>> Response code to my ACCEPT is 202 as expected. >>>>>>>>> >>>>>>>>> Here is my HTTP json message: >>>>>>>>> >>>>>>>>> {'type': 'ACCEPT', >>>>>>>>> 'framework_id': {'value': >>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, >>>>>>>>> 'accept': { >>>>>>>>> 'operations': [ >>>>>>>>> {'type': 'LAUNCH', >>>>>>>>> 'launch': {'container': { >>>>>>>>> 'docker': {'image': u'centos:latest', >>>>>>>>> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, >>>>>>>>> 'type': 1, >>>>>>>>> 'volumes': [ >>>>>>>>> {'host_path': u'/a/b', 'container_path': >>>>>>>>> u'/mnt/home', 'mode': 1}, >>>>>>>>> {'host_path': u'/a/b/c', 'container_path': >>>>>>>>> u'/mnt/go-docker', 'mode': 1}, >>>>>>>>> {'host_path': u'/b/c/d', 'container_path': >>>>>>>>> u'/mnt/god-data', 'mode': 2} >>>>>>>>> ] >>>>>>>>> }, >>>>>>>>> 'name': u'testr', >>>>>>>>> 'task_id': {'value': '128'}, >>>>>>>>> 'command': {'uris': [{'value': >>>>>>>>> u'/home/osallou/docker.tar.gz'}], 'value': >>>>>>> u'/mnt/go-docker/wrapper.sh'}, >>>>>>>>> 'slave_id': {'value': >>>>>>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, >>>>>>>>> 'resources': [ >>>>>>>>> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, >>>>>>>>> {'scalar': {'value': 2000}, 'type': 0, 'name': >> 'mem'} >>>>>>>>> ] >>>>>>>>> } # end launch >>>>>>>>> } # end operation >>>>>>>>> ], >>>>>>>>> 'offer_ids': [{'value': >>>>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> There could be an issue with my task definition, but as no error is >>>>>>>>> raised and I receive no UPDATE error message..... >>>>>>>>> >>>>>>>>> Any hint on how to debug this? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Olivier Sallou >>>>>>>>> IRISA / University of Rennes 1 >>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>>>>>>> Tel: 02.99.84.71.95 >>>>>>>>> >>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D >> 8438 >>>>>>>>> >>>>>>> -- >>>>>>> Olivier Sallou >>>>>>> IRISA / University of Rennes 1 >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>>>>> Tel: 02.99.84.71.95 >>>>>>> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>>>> >>>>>>> >>>>> -- >>>>> Olivier Sallou >>>>> IRISA / University of Rennes 1 >>>>> Campus de Beaulieu, 35000 RENNES - FRANCE >>>>> Tel: 02.99.84.71.95 >>>>> >>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>> >>>>> >>>>> >>> -- >>> Olivier Sallou >>> IRISA / University of Rennes 1 >>> Campus de Beaulieu, 35000 RENNES - FRANCE >>> Tel: 02.99.84.71.95 >>> >>> gpg key id: 4096R/326D8438 (keyring.debian.org < >> http://keyring.debian.org/>) >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >> -- Olivier Sallou IRISA / University of Rennes 1 Campus de Beaulieu, 35000 RENNES - FRANCE Tel: 02.99.84.71.95 gpg key id: 4096R/326D8438 (keyring.debian.org) Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
