Hi Anand, Did you file a JIRA for this? I saw that there is no message if the LAUNCH info do not have task info https://github.com/apache/mesos/blob/master/src/master/master.cpp#L3463-L3480 , then the master will decline the offer when launch task https://github.com/apache/mesos/blob/master/src/master/master.cpp#L3068
It is better to log some message if there is no task in launch info. Thanks, Guangya On Tue, Jun 7, 2016 at 11:46 PM, Anand Mazumdar <[email protected]> wrote: > Olivier, > > You are missing the “task_infos” key in your “ACCEPT” call. The master > treats “Accept” operations with no launch tasks as declining offers > implicitly. I would file a followup JIRA to ensure this is logged on the > master (if not so). > > An example correct JSON: > https://gist.github.com/hatred/7325d8a4afde607ecc0f376ab62d60eb < > https://gist.github.com/hatred/7325d8a4afde607ecc0f376ab62d60eb> > > -anand > > > On Jun 7, 2016, at 8:38 AM, Olivier Sallou <[email protected]> > wrote: > > > > > > > > On 06/07/2016 04:53 PM, Guangya Liu wrote: > >> So how many agent nodes are there in your cluster? If you continue > >> receiving offer but without getting UPDATE message, then it may be > caused > >> by that your task definition and the framework continually decline > offer. > > I have only one node (master/slave), for development. It worked fine > > with the python API. > > we see on master that it received the ACCEPT, and no DECLINE. However, > > as I receive no UPDATE, I suppose that mesos "drops" the ACCEPT (wrong > > task definition maybe), and sends new offers several seconds after I > > sent the ACCEPT. > >> > >> Can you please share your framework code here for the logic of "Event:: > >> OFFERS"? > > Code is available here: > > > > > https://bitbucket.org/osallou/go-docker/src/b1948063fb7f68fbc77f5de6b473d832a7dd36af/plugins/mesos.py?at=master&fileviewer=file-view-default > < > https://bitbucket.org/osallou/go-docker/src/b1948063fb7f68fbc77f5de6b473d832a7dd36af/plugins/mesos.py?at=master&fileviewer=file-view-default > > > > > > in method run of MesosThread, line 613 > > > > Code is a little complex, as it is a port of existing code using mesos > > python lib. > > > > Code related to HTTP is in development, so there may be further errors, > > but registration is fine as well as offer messages. > > > > I have added locally a debug print to show any message received by mesos > > (in case I would have received an other message indicating an error), > > but I received no other than offer and heartbeats. > > > > If Mesos see the ACCEPT message as it appears in logs, that it should > > either reject it (with a different status code than 202) or send an > > UPDATE error message if there is an error with my task definition. > > > > Olivier > >> > >> Thanks, > >> > >> Guangya > >> > >> On Tue, Jun 7, 2016 at 8:29 PM, Olivier Sallou <[email protected] > > > >> wrote: > >> > >>> > >>> On 06/07/2016 01:59 PM, Guangya Liu wrote: > >>>> I can see that your framework is now holding the offer, how did you > >>> launch > >>>> task? > >>> I execute an HTTP POST request in Python with json content-type: > >>> > >>> {'type': 'ACCEPT', > >>> 'framework_id': {'value': > u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, > >>> 'accept': { > >>> 'operations': [ > >>> {'type': 'LAUNCH', > >>> 'launch': {'container': { > >>> 'docker': {'image': u'centos:latest', > >>> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, > >>> 'type': 1, > >>> 'volumes': [ > >>> {'host_path': u'/a/b', 'container_path': > >>> u'/mnt/home', 'mode': 1}, > >>> {'host_path': u'/a/b/c', 'container_path': > >>> u'/mnt/go-docker', 'mode': 1}, > >>> {'host_path': u'/b/c/d', 'container_path': > >>> u'/mnt/god-data', 'mode': 2} > >>> ] > >>> }, > >>> 'name': u'testr', > >>> 'task_id': {'value': '128'}, > >>> 'command': {'uris': [{'value': > >>> u'/home/osallou/docker.tar.gz'}], 'value': > u'/mnt/go-docker/wrapper.sh'}, > >>> 'slave_id': {'value': > >>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, > >>> 'resources': [ > >>> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, > >>> {'scalar': {'value': 2000}, 'type': 0, 'name': 'mem'} > >>> ] > >>> } # end launch > >>> } # end operation > >>> ], > >>> 'offer_ids': [{'value': > u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] > >>> } > >>> } > >>> > >>> We can see that Mesos received the ACCEPT: > >>> > >>> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call for > >>> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave > >>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 > >>> (tifenn.irisa.fr) for framework > >>> > >>> > >>> and I continue to receive new offers, so "connection" is OK. I should > >>> receive an UPDATE message even if there is an error, but I receive none > >>> (I track/log all messages received, whatever the type). > >>> > >>> Olivier > >>> > >>>> Perhaps you can take a look at > >>>> https://github.com/apache/mesos/blob/master/src/cli/execute.cpp#L311 > >>> which > >>>> is an example framework using HTTP API > >>>> > >>>> Thanks, > >>>> > >>>> Guangya > >>>> > >>>> On Tue, Jun 7, 2016 at 7:19 PM, Olivier Sallou < > [email protected]> > >>>> wrote: > >>>> > >>>>> On 06/07/2016 12:25 PM, Guangya Liu wrote: > >>>>>> Olivier, > >>>>>> > >>>>>> For such case, seems there is sth wrong with your framework? can you > >>>>> please > >>>>>> run the following two commands and check the output? > >>>>> I don't think it is a framework issue, I receive offers, heartbeats > >>> etc... > >>>>> It is only at task creation step, when I have no rejection nor update > >>>>> message. > >>>>> > >>>>> It could be (certainly) an issue with the json task message I sent in > >>>>> the ACCEPT, but as there is no error, I have no way to understand > what's > >>>>> wrong with it. > >>>>>> curl "http://<mesos-master>:5050/master/frameworks" > 2>/dev/null|python > >>>>> -m > >>>>>> json.tool > >>>>> { > >>>>> "completed_frameworks": [], > >>>>> "frameworks": [ > >>>>> { > >>>>> "active": true, > >>>>> "capabilities": [], > >>>>> "checkpoint": false, > >>>>> "completed_tasks": [], > >>>>> "executors": [], > >>>>> "failover_timeout": 0.0, > >>>>> "hostname": "", > >>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", > >>>>> "name": "GoDocker HTTP Framework", > >>>>> "offered_resources": { > >>>>> "cpus": 4.0, > >>>>> "disk": 459470.0, > >>>>> "mem": 14898.0, > >>>>> "ports": "[31000-32000]" > >>>>> }, > >>>>> "offers": [ > >>>>> { > >>>>> "framework_id": > >>>>> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", > >>>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956-O0", > >>>>> "resources": { > >>>>> "cpus": 4.0, > >>>>> "disk": 459470.0, > >>>>> "mem": 14898.0, > >>>>> "ports": "[31000-32000]" > >>>>> }, > >>>>> "slave_id": > >>> "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0" > >>>>> } > >>>>> ], > >>>>> "registered_time": 1465298174.2483, > >>>>> "resources": { > >>>>> "cpus": 4.0, > >>>>> "disk": 459470.0, > >>>>> "mem": 14898.0, > >>>>> "ports": "[31000-32000]" > >>>>> }, > >>>>> "role": "*", > >>>>> "tasks": [], > >>>>> "unregistered_time": 0.0, > >>>>> "used_resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "user": "godocker_http_test", > >>>>> "webui_url": "" > >>>>> } > >>>>> ], > >>>>> "unregistered_frameworks": [] > >>>>> } > >>>>> > >>>>> > >>>>>> curl "http://<mesos-master>:5050/master/state" 2>/dev/null|python > -m > >>>>>> json.tool > >>>>> { > >>>>> "activated_slaves": 1.0, > >>>>> "build_date": "2016-04-14 15:44:54", > >>>>> "build_time": 1460648694.0, > >>>>> "build_user": "root", > >>>>> "completed_frameworks": [], > >>>>> "deactivated_slaves": 0.0, > >>>>> "elected_time": 1465298164.01165, > >>>>> "flags": { > >>>>> "allocation_interval": "1secs", > >>>>> "allocator": "HierarchicalDRF", > >>>>> "authenticate": "false", > >>>>> "authenticate_http": "false", > >>>>> "authenticate_slaves": "false", > >>>>> "authenticators": "crammd5", > >>>>> "authorizers": "local", > >>>>> "framework_sorter": "drf", > >>>>> "help": "false", > >>>>> "hostname_lookup": "true", > >>>>> "http_authenticators": "basic", > >>>>> "initialize_driver_logging": "true", > >>>>> "log_auto_initialize": "true", > >>>>> "log_dir": "/var/log/mesos", > >>>>> "logbufsecs": "0", > >>>>> "logging_level": "INFO", > >>>>> "max_completed_frameworks": "50", > >>>>> "max_completed_tasks_per_framework": "1000", > >>>>> "max_slave_ping_timeouts": "5", > >>>>> "port": "5050", > >>>>> "quiet": "false", > >>>>> "quorum": "1", > >>>>> "recovery_slave_removal_limit": "100%", > >>>>> "registry": "replicated_log", > >>>>> "registry_fetch_timeout": "1mins", > >>>>> "registry_store_timeout": "20secs", > >>>>> "registry_strict": "false", > >>>>> "root_submissions": "true", > >>>>> "slave_ping_timeout": "15secs", > >>>>> "slave_reregister_timeout": "10mins", > >>>>> "user_sorter": "drf", > >>>>> "version": "false", > >>>>> "webui_dir": "/usr/share/mesos/webui", > >>>>> "work_dir": "/var/lib/mesos", > >>>>> "zk": "zk://localhost:2181/mesos", > >>>>> "zk_session_timeout": "10secs" > >>>>> }, > >>>>> "frameworks": [ > >>>>> { > >>>>> "active": true, > >>>>> "capabilities": [], > >>>>> "checkpoint": false, > >>>>> "completed_tasks": [], > >>>>> "executors": [], > >>>>> "failover_timeout": 0.0, > >>>>> "hostname": "", > >>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0021", > >>>>> "name": "GoDocker HTTP Framework", > >>>>> "offered_resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "offers": [], > >>>>> "registered_time": 1465298174.2483, > >>>>> "resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "role": "*", > >>>>> "tasks": [], > >>>>> "unregistered_time": 0.0, > >>>>> "used_resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "user": "godocker_http_test", > >>>>> "webui_url": "" > >>>>> } > >>>>> ], > >>>>> "git_sha": "555db235a34afbb9fb49940376cc33a66f1f85f0", > >>>>> "git_tag": "0.28.1", > >>>>> "hostname": "tifenn.irisa.fr", > >>>>> "id": "1f1486e3-43ee-44c5-b073-82a901add956", > >>>>> "leader": "[email protected]:5050", > >>>>> "log_dir": "/var/log/mesos", > >>>>> "orphan_tasks": [], > >>>>> "pid": "[email protected]:5050", > >>>>> "slaves": [ > >>>>> { > >>>>> "active": true, > >>>>> "attributes": { > >>>>> "hostname": "127.0.0.1" > >>>>> }, > >>>>> "hostname": "tifenn.irisa.fr", > >>>>> "id": "e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0", > >>>>> "offered_resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "pid": "slave(1)@127.0.1.1:5051", > >>>>> "registered_time": 1465298164.37517, > >>>>> "reregistered_time": 1465298164.37526, > >>>>> "reserved_resources": {}, > >>>>> "resources": { > >>>>> "cpus": 4.0, > >>>>> "disk": 459470.0, > >>>>> "mem": 14898.0, > >>>>> "ports": "[31000-32000]" > >>>>> }, > >>>>> "unreserved_resources": { > >>>>> "cpus": 4.0, > >>>>> "disk": 459470.0, > >>>>> "mem": 14898.0, > >>>>> "ports": "[31000-32000]" > >>>>> }, > >>>>> "used_resources": { > >>>>> "cpus": 0.0, > >>>>> "disk": 0.0, > >>>>> "mem": 0.0 > >>>>> }, > >>>>> "version": "0.28.1" > >>>>> } > >>>>> ], > >>>>> "start_time": 1465298159.26321, > >>>>> "unregistered_frameworks": [], > >>>>> "version": "0.28.1" > >>>>> } > >>>>> > >>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Guangya > >>>>>> > >>>>>> On Tue, Jun 7, 2016 at 6:04 PM, Olivier Sallou < > >>> [email protected]> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi, > >>>>>>> I am trying to switch from Python to HTTP API. I use mesos 0.28.1 > >>>>>>> > >>>>>>> I could create framework to register, receive offers etc... but I > >>> have > >>>>>>> an issue accepting offers. > >>>>>>> > >>>>>>> I send my ACCEPT message but I do not receive any UPDATE message, > only > >>>>>>> new offers and hearbeat messages. > >>>>>>> > >>>>>>> On mesos master logs I see: > >>>>>>> > >>>>>>> I0607 11:45:15.873184 14896 http.cpp:312] HTTP POST for > >>>>>>> /master/api/v1/scheduler from 127.0.0.1:38298 with > >>>>>>> User-Agent='python-requests/2.9.1' > >>>>>>> I0607 11:45:15.873584 14896 master.cpp:3104] Processing ACCEPT call > >>> for > >>>>>>> offers: [ e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28 ] on slave > >>>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0 at slave(1)@127.0.1.1:5051 > >>>>>>> (tifenn.irisa.fr) for framework > >>>>>>> e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020 (GoDocker HTTP Framework) > >>>>>>> > >>>>>>> There is a "Processing ACCEPT" and no error, but my task is not > ran on > >>>>>>> mesos. > >>>>>>> No error on slave either. > >>>>>>> > >>>>>>> Response code to my ACCEPT is 202 as expected. > >>>>>>> > >>>>>>> Here is my HTTP json message: > >>>>>>> > >>>>>>> {'type': 'ACCEPT', > >>>>>>> 'framework_id': {'value': > >>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-0020'}, > >>>>>>> 'accept': { > >>>>>>> 'operations': [ > >>>>>>> {'type': 'LAUNCH', > >>>>>>> 'launch': {'container': { > >>>>>>> 'docker': {'image': u'centos:latest', > >>>>>>> 'force_pull_image': True, 'port_mappings': [], 'network': 2}, > >>>>>>> 'type': 1, > >>>>>>> 'volumes': [ > >>>>>>> {'host_path': u'/a/b', 'container_path': > >>>>>>> u'/mnt/home', 'mode': 1}, > >>>>>>> {'host_path': u'/a/b/c', 'container_path': > >>>>>>> u'/mnt/go-docker', 'mode': 1}, > >>>>>>> {'host_path': u'/b/c/d', 'container_path': > >>>>>>> u'/mnt/god-data', 'mode': 2} > >>>>>>> ] > >>>>>>> }, > >>>>>>> 'name': u'testr', > >>>>>>> 'task_id': {'value': '128'}, > >>>>>>> 'command': {'uris': [{'value': > >>>>>>> u'/home/osallou/docker.tar.gz'}], 'value': > >>>>> u'/mnt/go-docker/wrapper.sh'}, > >>>>>>> 'slave_id': {'value': > >>>>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-S0'}, > >>>>>>> 'resources': [ > >>>>>>> {'scalar': {'value': 1}, 'type': 0, 'name': 'cpus'}, > >>>>>>> {'scalar': {'value': 2000}, 'type': 0, 'name': > 'mem'} > >>>>>>> ] > >>>>>>> } # end launch > >>>>>>> } # end operation > >>>>>>> ], > >>>>>>> 'offer_ids': [{'value': > >>>>> u'e303a1f0-4e7c-4c32-aafc-8707ea2b2718-O28'}] > >>>>>>> } > >>>>>>> } > >>>>>>> > >>>>>>> There could be an issue with my task definition, but as no error is > >>>>>>> raised and I receive no UPDATE error message..... > >>>>>>> > >>>>>>> Any hint on how to debug this? > >>>>>>> > >>>>>>> Thanks > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Olivier Sallou > >>>>>>> IRISA / University of Rennes 1 > >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE > >>>>>>> Tel: 02.99.84.71.95 > >>>>>>> > >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) > >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D > 8438 > >>>>>>> > >>>>>>> > >>>>> -- > >>>>> Olivier Sallou > >>>>> IRISA / University of Rennes 1 > >>>>> Campus de Beaulieu, 35000 RENNES - FRANCE > >>>>> Tel: 02.99.84.71.95 > >>>>> > >>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) > >>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > >>>>> > >>>>> > >>> -- > >>> Olivier Sallou > >>> IRISA / University of Rennes 1 > >>> Campus de Beaulieu, 35000 RENNES - FRANCE > >>> Tel: 02.99.84.71.95 > >>> > >>> gpg key id: 4096R/326D8438 (keyring.debian.org) > >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > >>> > >>> > >>> > > > > -- > > Olivier Sallou > > IRISA / University of Rennes 1 > > Campus de Beaulieu, 35000 RENNES - FRANCE > > Tel: 02.99.84.71.95 > > > > gpg key id: 4096R/326D8438 (keyring.debian.org < > http://keyring.debian.org/>) > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > >
