On 11/8/18 11:58 PM, Gilbert Song wrote: > > Hi Olivier and Adam, > > This issue is already fixed > by https://issues.apache.org/jira/browse/MESOS-8907 which was merged > and backported 17 days ago. Please try it out and let me know.
My code is already at latest (HEAD) and include http/1.1 fix. In log, we can see by the way that it uses 1.1: 80fc2079-ba14-454b-8276-79fae090f8b3-0000 failed to start: Failed to decode HTTP responses: Decoding failed HTTP/1.1 200 OK Content-Type: application/json Date: Wed, 07 Nov 2018 09:45:54 GMT Transfer-Encoding: chunked ... I have no proxy in between, and uses curl 7.58.0 (Ubuntu 18.04). Seems indeed linked to curl version When error is displayed in docker.cpp, body looks indeed strange .... Transfer-Encoding: chunked Strict-Transport-Security: max-age=31536000 105d {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp......."} 0 {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp......."} After the *0* which should mark the end of chunked data, there is additional data (the same json token info). I think http parser fails at parsing when reaching the *0* and finding other data (not expecting it) It tries to fetch this URL: https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/centos:pull If I try the same URL with an older curl version (7.19) HTTP/1.1 200 OK .... Transfer-Encoding: chunked Strict-Transport-Security: max-age=31536000 fb3 {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6..."} 0 no duplicate token here Olivier > > More background contexts: > > 1. http-parse does not support > http/2: https://issues.apache.org/jira/browse/MESOS-8368 > 2. curl enable http/2 by default after curl 7.47 > > - Gilbert > > On Thu, Nov 8, 2018 at 2:33 PM Cecile, Adam <adam.cec...@hitec.lu > <mailto:adam.cec...@hitec.lu>> wrote: > > http-parser library is not updated anymore and does not support > HTTP/2 so it's a dead end anyway > > Le 8 novembre 2018 10:17:20 GMT+01:00, Olivier Sallou > <olivier.sal...@irisa.fr <mailto:olivier.sal...@irisa.fr>> a écrit : > > > ----- Mail original ----- > > De: "Olivier Sallou" <olivier.sal...@irisa.fr > <mailto:olivier.sal...@irisa.fr>> À: user@mesos.apache.org > <mailto:user@mesos.apache.org> Envoyé: Mercredi 7 Novembre > 2018 12:40:24 Objet: Re: mesos containerizer issue with > v1.8.0: HTTP response Decoding failed > > > On 11/7/18 11:54 AM, Cecile, Adam wrote: > > Hi, You might be hitting the same bug as I did (no > HTTP/2 support in code pulling images for Mesos). > https://issues.apache.org/jira/browse/MESOS-9364 > > adding some logs as suggested in your issue, error code is > different, getting "invalid constant string" error from > http_parser error: XX(INVALID_CONSTANT, "invalid > constant string") > > > > with additional debug, problem is related to http parser vs received > answer from dockerhub. If I remove control on parsed data length vs body > length it works nicely (for test/debug only....), so it may be a pb of > response http compliance... > Anyway, this issue prevents download of docker images when using > unified containerizer. As I am using code from master branch (latest), I can > only hope it will be fixed before next release.... > > > > My report also include some code you can add in C++ > code of the fetcher to retrieve the actual message > coming from the http response parser library. Regards, > Adam. On 11/7/18 11:42 AM, Adam Cecile wrote: > > On 11/7/18 10:48 AM, Olivier Sallou wrote: > > On 11/7/18 10:38 AM, Olivier Sallou wrote: > > Hi, I installed mesos from source. It > works fine with docker containerizer. > Howerver it fails with unified > containerizer at container start. It used > to work on a previous (older release) > install. In the meanwhile, some system > libs etc.. have been upgraded. In logs I > have the following: I1107 09:32:48.707176 > 31983 containerizer.cpp:1280] Starting > container > 28f07a61-676a-4876-aae4-73598de90aae E1107 > 09:32:49.683372 31986 slave.cpp:6168] > Container > '28f07a61-676a-4876-aae4-73598de90aae' for > executor '1-0' of framework > 80fc2079-ba14-454b-8276-79fae090f8b3-0000 > failed to start: Failed to decode HTTP > responses: Decoding failed HTTP/1.1 200 OK > Content-Type: application/json Date: Wed, > 07 Nov 2018 08:32:46 GMT > Transfer-Encoding: chunked > Strict-Transport-Security: > max-age=31536000 105c > {"token":"eyJhbGciOiJSUzI1NiIsInR5cC....... > Logs do not show the destination of the > http request (a pull on docker hub ? a > request to master ? ...) > > I could increase some slave logging, and HTTP > failure occurs with pulling of image: > I1107 10:45:56.689092 31987 > registry_puller.cpp:286] Pulling image > 'library/centos:latest' from > > 'docker-manifest://registry-1.docker.io:443library/centos?latest#https' > to '/tmp/mesos/store/docker/staging/99WUh3' > E1107 10:45:57.634601 31987 > slave.cpp:6168] Container > '48ea5811-3f97-41c1-b1a5-9a4416552545' for > executor '6-0' of framework > 80fc2079-ba14-454b-8276-79fae090f8b3-0000 > failed to start: Failed to decode HTTP > responses: Decoding failed HTTP/1.1 200 > OK Content-Type: application/json > Date: Wed, 07 Nov 2018 09:45:54 GMT > Transfer-Encoding: chunked > Strict-Transport-Security: max-age=31536000 > .... so it seems there is an issue with > mesos unified management with docker hub. > Could it be related to libcurl version? > (libcurl4 on my system). Is specific setup > needed? > > any idea on what could be wrong ? or how > to get more debug info ? Thanks Olivier > > -- Olivier Sallou Univ Rennes, Inria, CNRS, IRISA Irisa, > Campus de Beaulieu F-35042 RENNES - FRANCE Tel: > 02.99.84.71.95 gpg key id: 4096R/326D8438 > (keyring.debian.org <http://keyring.debian.org>) Key > fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D > 8438 > > > -- > Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez > excuser ma brièveté. > -- gpg key id: 4096R/326D8438 (keyring.debian.org) Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438