On 11/9/18 11:44 AM, Olivier Sallou wrote: > > > On 11/8/18 11:58 PM, Gilbert Song wrote: >> >> Hi Olivier and Adam, >> >> This issue is already fixed >> by https://issues.apache.org/jira/browse/MESOS-8907 which was merged >> and backported 17 days ago. Please try it out and let me know. > > > My code is already at latest (HEAD) and include http/1.1 fix. > > In log, we can see by the way that it uses 1.1: > > > 80fc2079-ba14-454b-8276-79fae090f8b3-0000 failed to start: Failed > to decode HTTP responses: Decoding failed > HTTP/1.1 200 OK > > Content-Type: application/json > Date: Wed, 07 Nov 2018 09:45:54 GMT > Transfer-Encoding: chunked > > ... > > > I have no proxy in between, and uses curl 7.58.0 (Ubuntu 18.04). Seems > indeed linked to curl version >
confirming curl issue (https://github.com/curl/curl/issues/2303), there is a problem with raw option with chunked data, fixed in version 7.59.0 as http parser finds more data than expected, it fails... Olivier > > When error is displayed in docker.cpp, body looks indeed strange > > .... > > Transfer-Encoding: chunked > Strict-Transport-Security: max-age=31536000 > > 105d > {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp......."} > > 0 > > {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp......."} > > > After the *0* which should mark the end of chunked data, there is > additional data (the same json token info). I think http parser fails > at parsing when reaching the *0* and finding other data (not expecting it) > > > It tries to fetch this URL: > https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/centos:pull > > > If I try the same URL with an older curl version (7.19) > > > HTTP/1.1 200 OK > > .... > > Transfer-Encoding: chunked > Strict-Transport-Security: max-age=31536000 > > fb3 > {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6..."} > > 0 > > > no duplicate token here > > > Olivier > > > >> >> More background contexts: >> >> 1. http-parse does not support >> http/2: https://issues.apache.org/jira/browse/MESOS-8368 >> 2. curl enable http/2 by default after curl 7.47 >> >> - Gilbert >> >> On Thu, Nov 8, 2018 at 2:33 PM Cecile, Adam <adam.cec...@hitec.lu >> <mailto:adam.cec...@hitec.lu>> wrote: >> >> http-parser library is not updated anymore and does not support >> HTTP/2 so it's a dead end anyway >> >> Le 8 novembre 2018 10:17:20 GMT+01:00, Olivier Sallou >> <olivier.sal...@irisa.fr <mailto:olivier.sal...@irisa.fr>> a écrit : >> >> ----- Mail original ----- >> >> De: "Olivier Sallou" <olivier.sal...@irisa.fr >> <mailto:olivier.sal...@irisa.fr>> À: >> user@mesos.apache.org <mailto:user@mesos.apache.org> >> Envoyé: Mercredi 7 Novembre 2018 12:40:24 Objet: Re: >> mesos containerizer issue with v1.8.0: HTTP response >> Decoding failed >> >> >> On 11/7/18 11:54 AM, Cecile, Adam wrote: >> >> Hi, You might be hitting the same bug as I did (no >> HTTP/2 support in code pulling images for Mesos). >> https://issues.apache.org/jira/browse/MESOS-9364 >> >> adding some logs as suggested in your issue, error code >> is different, getting "invalid constant string" error >> from http_parser error: XX(INVALID_CONSTANT, >> "invalid constant string") >> >> >> >> with additional debug, problem is related to http parser vs received >> answer from dockerhub. If I remove control on parsed data length vs body >> length it works nicely (for test/debug only....), so it may be a pb of >> response http compliance... >> Anyway, this issue prevents download of docker images when using >> unified containerizer. As I am using code from master branch (latest), I can >> only hope it will be fixed before next release.... >> >> >> >> My report also include some code you can add in C++ >> code of the fetcher to retrieve the actual message >> coming from the http response parser library. >> Regards, Adam. On 11/7/18 11:42 AM, Adam Cecile wrote: >> >> On 11/7/18 10:48 AM, Olivier Sallou wrote: >> >> On 11/7/18 10:38 AM, Olivier Sallou wrote: >> >> Hi, I installed mesos from source. It >> works fine with docker containerizer. >> Howerver it fails with unified >> containerizer at container start. It used >> to work on a previous (older release) >> install. In the meanwhile, some system >> libs etc.. have been upgraded. In logs I >> have the following: I1107 09:32:48.707176 >> 31983 containerizer.cpp:1280] Starting >> container >> 28f07a61-676a-4876-aae4-73598de90aae >> E1107 09:32:49.683372 31986 >> slave.cpp:6168] Container >> '28f07a61-676a-4876-aae4-73598de90aae' >> for executor '1-0' of framework >> 80fc2079-ba14-454b-8276-79fae090f8b3-0000 >> failed to start: Failed to decode HTTP >> responses: Decoding failed HTTP/1.1 200 >> OK Content-Type: application/json Date: >> Wed, 07 Nov 2018 08:32:46 GMT >> Transfer-Encoding: chunked >> Strict-Transport-Security: >> max-age=31536000 105c >> {"token":"eyJhbGciOiJSUzI1NiIsInR5cC....... >> Logs do not show the destination of the >> http request (a pull on docker hub ? a >> request to master ? ...) >> >> I could increase some slave logging, and HTTP >> failure occurs with pulling of image: >> I1107 10:45:56.689092 31987 >> registry_puller.cpp:286] Pulling image >> 'library/centos:latest' from >> >> 'docker-manifest://registry-1.docker.io:443library/centos?latest#https' >> to '/tmp/mesos/store/docker/staging/99WUh3' >> E1107 10:45:57.634601 31987 >> slave.cpp:6168] Container >> '48ea5811-3f97-41c1-b1a5-9a4416552545' for >> executor '6-0' of framework >> 80fc2079-ba14-454b-8276-79fae090f8b3-0000 >> failed to start: Failed to decode HTTP >> responses: Decoding failed HTTP/1.1 200 >> OK Content-Type: application/json >> Date: Wed, 07 Nov 2018 09:45:54 GMT >> Transfer-Encoding: chunked >> Strict-Transport-Security: max-age=31536000 >> .... so it seems there is an issue with >> mesos unified management with docker hub. >> Could it be related to libcurl version? >> (libcurl4 on my system). Is specific setup >> needed? >> >> any idea on what could be wrong ? or how >> to get more debug info ? Thanks Olivier >> >> -- Olivier Sallou Univ Rennes, Inria, CNRS, IRISA Irisa, >> Campus de Beaulieu F-35042 RENNES - FRANCE Tel: >> 02.99.84.71.95 gpg key id: 4096R/326D8438 >> (keyring.debian.org <http://keyring.debian.org>) Key >> fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB >> 326D 8438 >> >> >> -- >> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez >> excuser ma brièveté. >> > -- > > > gpg key id: 4096R/326D8438 (keyring.debian.org) > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > -- Olivier Sallou Univ Rennes, Inria, CNRS, IRISA Irisa, Campus de Beaulieu F-35042 RENNES - FRANCE Tel: 02.99.84.71.95 gpg key id: 4096R/326D8438 (keyring.debian.org) Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
