On 11/8/18 11:58 PM, Gilbert Song wrote:
>
> Hi Olivier and Adam,
>
> This issue is already fixed
> by https://issues.apache.org/jira/browse/MESOS-8907 which was merged
> and backported 17 days ago. Please try it out and let me know.
My code is already at latest (HEAD) and include http/1.1 fix.
In log, we can see by the way that it uses 1.1:
80fc2079-ba14-454b-8276-79fae090f8b3-0000 failed to start: Failed to
decode HTTP responses: Decoding failed
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 07 Nov 2018 09:45:54 GMT
Transfer-Encoding: chunked
...
I have no proxy in between, and uses curl 7.58.0 (Ubuntu 18.04). Seems
indeed linked to curl version
When error is displayed in docker.cpp, body looks indeed strange
....
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
105d
{"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp......."}
0
{"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp......."}
After the *0* which should mark the end of chunked data, there is
additional data (the same json token info). I think http parser fails at
parsing when reaching the *0* and finding other data (not expecting it)
It tries to fetch this URL:
https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/centos:pull
If I try the same URL with an older curl version (7.19)
HTTP/1.1 200 OK
....
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
fb3
{"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6..."}
0
no duplicate token here
Olivier
>
> More background contexts:
>
> 1. http-parse does not support
> http/2: https://issues.apache.org/jira/browse/MESOS-8368
> 2. curl enable http/2 by default after curl 7.47
>
> - Gilbert
>
> On Thu, Nov 8, 2018 at 2:33 PM Cecile, Adam <[email protected]
> <mailto:[email protected]>> wrote:
>
> http-parser library is not updated anymore and does not support
> HTTP/2 so it's a dead end anyway
>
> Le 8 novembre 2018 10:17:20 GMT+01:00, Olivier Sallou
> <[email protected] <mailto:[email protected]>> a écrit :
>
>
> ----- Mail original -----
>
> De: "Olivier Sallou" <[email protected]
> <mailto:[email protected]>> À: [email protected]
> <mailto:[email protected]> Envoyé: Mercredi 7 Novembre
> 2018 12:40:24 Objet: Re: mesos containerizer issue with
> v1.8.0: HTTP response Decoding failed
>
>
> On 11/7/18 11:54 AM, Cecile, Adam wrote:
>
> Hi, You might be hitting the same bug as I did (no
> HTTP/2 support in code pulling images for Mesos).
> https://issues.apache.org/jira/browse/MESOS-9364
>
> adding some logs as suggested in your issue, error code is
> different, getting "invalid constant string" error from
> http_parser error: XX(INVALID_CONSTANT, "invalid
> constant string")
>
>
>
> with additional debug, problem is related to http parser vs received
> answer from dockerhub. If I remove control on parsed data length vs body
> length it works nicely (for test/debug only....), so it may be a pb of
> response http compliance...
> Anyway, this issue prevents download of docker images when using
> unified containerizer. As I am using code from master branch (latest), I can
> only hope it will be fixed before next release....
>
>
>
> My report also include some code you can add in C++
> code of the fetcher to retrieve the actual message
> coming from the http response parser library. Regards,
> Adam. On 11/7/18 11:42 AM, Adam Cecile wrote:
>
> On 11/7/18 10:48 AM, Olivier Sallou wrote:
>
> On 11/7/18 10:38 AM, Olivier Sallou wrote:
>
> Hi, I installed mesos from source. It
> works fine with docker containerizer.
> Howerver it fails with unified
> containerizer at container start. It used
> to work on a previous (older release)
> install. In the meanwhile, some system
> libs etc.. have been upgraded. In logs I
> have the following: I1107 09:32:48.707176
> 31983 containerizer.cpp:1280] Starting
> container
> 28f07a61-676a-4876-aae4-73598de90aae E1107
> 09:32:49.683372 31986 slave.cpp:6168]
> Container
> '28f07a61-676a-4876-aae4-73598de90aae' for
> executor '1-0' of framework
> 80fc2079-ba14-454b-8276-79fae090f8b3-0000
> failed to start: Failed to decode HTTP
> responses: Decoding failed HTTP/1.1 200 OK
> Content-Type: application/json Date: Wed,
> 07 Nov 2018 08:32:46 GMT
> Transfer-Encoding: chunked
> Strict-Transport-Security:
> max-age=31536000 105c
> {"token":"eyJhbGciOiJSUzI1NiIsInR5cC.......
> Logs do not show the destination of the
> http request (a pull on docker hub ? a
> request to master ? ...)
>
> I could increase some slave logging, and HTTP
> failure occurs with pulling of image:
> I1107 10:45:56.689092 31987
> registry_puller.cpp:286] Pulling image
> 'library/centos:latest' from
>
> 'docker-manifest://registry-1.docker.io:443library/centos?latest#https'
> to '/tmp/mesos/store/docker/staging/99WUh3'
> E1107 10:45:57.634601 31987
> slave.cpp:6168] Container
> '48ea5811-3f97-41c1-b1a5-9a4416552545' for
> executor '6-0' of framework
> 80fc2079-ba14-454b-8276-79fae090f8b3-0000
> failed to start: Failed to decode HTTP
> responses: Decoding failed HTTP/1.1 200
> OK Content-Type: application/json
> Date: Wed, 07 Nov 2018 09:45:54 GMT
> Transfer-Encoding: chunked
> Strict-Transport-Security: max-age=31536000
> .... so it seems there is an issue with
> mesos unified management with docker hub.
> Could it be related to libcurl version?
> (libcurl4 on my system). Is specific setup
> needed?
>
> any idea on what could be wrong ? or how
> to get more debug info ? Thanks Olivier
>
> -- Olivier Sallou Univ Rennes, Inria, CNRS, IRISA Irisa,
> Campus de Beaulieu F-35042 RENNES - FRANCE Tel:
> 02.99.84.71.95 gpg key id: 4096R/326D8438
> (keyring.debian.org <http://keyring.debian.org>) Key
> fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D
> 8438
>
>
> --
> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez
> excuser ma brièveté.
>
--
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
