Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
On 11/9/18 11:44 AM, Olivier Sallou wrote:
>
>
> On 11/8/18 11:58 PM, Gilbert Song wrote:
>>
>> Hi Olivier and Adam,
>>
>> This issue is already fixed
>> by https://issues.apache.org/jira/browse/MESOS-8907 which was merged
>> and backported 17 days ago. Please try it out and let me know.
>
>
> My code is already at latest (HEAD) and include http/1.1 fix.
>
> In log, we can see by the way that it uses 1.1:
>
>
> 80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed
> to decode HTTP responses: Decoding failed
> HTTP/1.1 200 OK
>
> Content-Type: application/json
> Date: Wed, 07 Nov 2018 09:45:54 GMT
> Transfer-Encoding: chunked
>
> ...
>
>
> I have no proxy in between, and uses curl 7.58.0 (Ubuntu 18.04). Seems
> indeed linked to curl version
>
confirming curl issue (https://github.com/curl/curl/issues/2303), there
is a problem with raw option with chunked data, fixed in version 7.59.0
as http parser finds more data than expected, it fails...
Olivier
>
> When error is displayed in docker.cpp, body looks indeed strange
>
>
>
> Transfer-Encoding: chunked
> Strict-Transport-Security: max-age=31536000
>
> 105d
> {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp..."}
>
> 0
>
> {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp..."}
>
>
> After the *0* which should mark the end of chunked data, there is
> additional data (the same json token info). I think http parser fails
> at parsing when reaching the *0* and finding other data (not expecting it)
>
>
> It tries to fetch this URL:
> https://auth.docker.io/token?service=registry.docker.io=repository:library/centos:pull
>
>
> If I try the same URL with an older curl version (7.19)
>
>
> HTTP/1.1 200 OK
>
>
>
> Transfer-Encoding: chunked
> Strict-Transport-Security: max-age=31536000
>
> fb3
> {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6..."}
>
> 0
>
>
> no duplicate token here
>
>
> Olivier
>
>
>
>>
>> More background contexts:
>>
>> 1. http-parse does not support
>> http/2: https://issues.apache.org/jira/browse/MESOS-8368
>> 2. curl enable http/2 by default after curl 7.47
>>
>> - Gilbert
>>
>> On Thu, Nov 8, 2018 at 2:33 PM Cecile, Adam > <mailto:adam.cec...@hitec.lu>> wrote:
>>
>> http-parser library is not updated anymore and does not support
>> HTTP/2 so it's a dead end anyway
>>
>> Le 8 novembre 2018 10:17:20 GMT+01:00, Olivier Sallou
>> mailto:olivier.sal...@irisa.fr>> a écrit :
>>
>> - Mail original -
>>
>> De: "Olivier Sallou" > <mailto:olivier.sal...@irisa.fr>> À:
>> user@mesos.apache.org <mailto:user@mesos.apache.org>
>> Envoyé: Mercredi 7 Novembre 2018 12:40:24 Objet: Re:
>> mesos containerizer issue with v1.8.0: HTTP response
>> Decoding failed
>>
>>
>> On 11/7/18 11:54 AM, Cecile, Adam wrote:
>>
>> Hi, You might be hitting the same bug as I did (no
>> HTTP/2 support in code pulling images for Mesos).
>> https://issues.apache.org/jira/browse/MESOS-9364
>>
>> adding some logs as suggested in your issue, error code
>> is different, getting "invalid constant string" error
>> from http_parser error: XX(INVALID_CONSTANT,
>> "invalid constant string")
>>
>>
>>
>> with additional debug, problem is related to http parser vs received
>> answer from dockerhub. If I remove control on parsed data length vs body
>> length it works nicely (for test/debug only), so it may be a pb of
>> response http compliance...
>> Anyway, this issue prevents download of docker images when using
>> unified containerizer. As I am using code from master branch (latest), I can
>> only hope it will be fixed before next release
>>
>>
>>
>> My report also include some code you can add in C++
>> code of the fetcher to retrieve the actual message
>> coming from the http response parser library.
>> Regards, Adam. On 11/7/18 11:42 AM, Adam Cecile wrote:
>>
>> On 11/7/18 10:48 AM, Olivier Sallou wrote:
>>
>>
Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
On 11/8/18 11:58 PM, Gilbert Song wrote:
>
> Hi Olivier and Adam,
>
> This issue is already fixed
> by https://issues.apache.org/jira/browse/MESOS-8907 which was merged
> and backported 17 days ago. Please try it out and let me know.
My code is already at latest (HEAD) and include http/1.1 fix.
In log, we can see by the way that it uses 1.1:
80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
decode HTTP responses: Decoding failed
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 07 Nov 2018 09:45:54 GMT
Transfer-Encoding: chunked
...
I have no proxy in between, and uses curl 7.58.0 (Ubuntu 18.04). Seems
indeed linked to curl version
When error is displayed in docker.cpp, body looks indeed strange
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
105d
{"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp..."}
0
{"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6Ikp..."}
After the *0* which should mark the end of chunked data, there is
additional data (the same json token info). I think http parser fails at
parsing when reaching the *0* and finding other data (not expecting it)
It tries to fetch this URL:
https://auth.docker.io/token?service=registry.docker.io=repository:library/centos:pull
If I try the same URL with an older curl version (7.19)
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
fb3
{"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6..."}
0
no duplicate token here
Olivier
>
> More background contexts:
>
> 1. http-parse does not support
> http/2: https://issues.apache.org/jira/browse/MESOS-8368
> 2. curl enable http/2 by default after curl 7.47
>
> - Gilbert
>
> On Thu, Nov 8, 2018 at 2:33 PM Cecile, Adam <mailto:adam.cec...@hitec.lu>> wrote:
>
> http-parser library is not updated anymore and does not support
> HTTP/2 so it's a dead end anyway
>
> Le 8 novembre 2018 10:17:20 GMT+01:00, Olivier Sallou
> mailto:olivier.sal...@irisa.fr>> a écrit :
>
>
> - Mail original -
>
> De: "Olivier Sallou" <mailto:olivier.sal...@irisa.fr>> À: user@mesos.apache.org
> <mailto:user@mesos.apache.org> Envoyé: Mercredi 7 Novembre
> 2018 12:40:24 Objet: Re: mesos containerizer issue with
> v1.8.0: HTTP response Decoding failed
>
>
> On 11/7/18 11:54 AM, Cecile, Adam wrote:
>
> Hi, You might be hitting the same bug as I did (no
> HTTP/2 support in code pulling images for Mesos).
> https://issues.apache.org/jira/browse/MESOS-9364
>
> adding some logs as suggested in your issue, error code is
> different, getting "invalid constant string" error from
> http_parser error: XX(INVALID_CONSTANT, "invalid
> constant string")
>
>
>
> with additional debug, problem is related to http parser vs received
> answer from dockerhub. If I remove control on parsed data length vs body
> length it works nicely (for test/debug only), so it may be a pb of
> response http compliance...
> Anyway, this issue prevents download of docker images when using
> unified containerizer. As I am using code from master branch (latest), I can
> only hope it will be fixed before next release
>
>
>
> My report also include some code you can add in C++
> code of the fetcher to retrieve the actual message
> coming from the http response parser library. Regards,
> Adam. On 11/7/18 11:42 AM, Adam Cecile wrote:
>
> On 11/7/18 10:48 AM, Olivier Sallou wrote:
>
> On 11/7/18 10:38 AM, Olivier Sallou wrote:
>
> Hi, I installed mesos from source. It
> works fine with docker containerizer.
> Howerver it fails with unified
> containerizer at container start. It used
> to work on a previous (older release)
> install. In the meanwhile, some system
> libs etc.. have been upgraded. In logs I
> have the following: I1107 09:32:48.707176
> 31983 containerizer.cpp:1280] Starting
> container
> 28f07a61-676a-4876-aae4-73598de90aae E1107
> 0
Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
Hi,
Thanks for the feedback, I missed it indeed (but we're running an older version
anyway).
Still, it's not really a fix but a temporary workaround: for instance NGINX
cannot share a single socket between 2 and 1.1 so you'll get bug reports coming
from private registry owners very soon.
Adam
Le 8 novembre 2018 23:58:11 GMT+01:00, Gilbert Song a
écrit :
Hi Olivier and Adam,
This issue is already fixed by https://issues.apache.org/jira/browse/MESOS-8907
which was merged and backported 17 days ago. Please try it out and let me know.
More background contexts:
1. http-parse does not support http/2:
https://issues.apache.org/jira/browse/MESOS-8368
2. curl enable http/2 by default after curl 7.47
- Gilbert
On Thu, Nov 8, 2018 at 2:33 PM Cecile, Adam
mailto:adam.cec...@hitec.lu>> wrote:
http-parser library is not updated anymore and does not support HTTP/2 so it's
a dead end anyway
Le 8 novembre 2018 10:17:20 GMT+01:00, Olivier Sallou
mailto:olivier.sal...@irisa.fr>> a écrit :
- Mail original -
De: "Olivier Sallou" mailto:olivier.sal...@irisa.fr>>
À: user@mesos.apache.org<mailto:user@mesos.apache.org>
Envoyé: Mercredi 7 Novembre 2018 12:40:24
Objet: Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
On 11/7/18 11:54 AM, Cecile, Adam wrote:
Hi,
You might be hitting the same bug as I did (no HTTP/2 support in code
pulling images for Mesos).
https://issues.apache.org/jira/browse/MESOS-9364
adding some logs as suggested in your issue, error code is different,
getting "invalid constant string" error from http_parser error:
XX(INVALID_CONSTANT, "invalid constant string")
with additional debug, problem is related to http parser vs received answer
from dockerhub. If I remove control on parsed data length vs body length it
works nicely (for test/debug only), so it may be a pb of response http
compliance...
Anyway, this issue prevents download of docker images when using unified
containerizer. As I am using code from master branch (latest), I can only hope
it will be fixed before next release
My report also include some code you can add in C++ code of the fetcher
to retrieve the actual message coming from the http response parser library.
Regards, Adam.
On 11/7/18 11:42 AM, Adam Cecile wrote:
On 11/7/18 10:48 AM, Olivier Sallou wrote:
On 11/7/18 10:38 AM, Olivier Sallou wrote:
Hi,
I installed mesos from source. It works fine with docker containerizer.
Howerver it fails with unified containerizer at container start.
It used to work on a previous (older release) install. In the
meanwhile,
some system libs etc.. have been upgraded.
In logs I have the following:
I1107 09:32:48.707176 31983 containerizer.cpp:1280] Starting container
28f07a61-676a-4876-aae4-73598de90aae
E1107 09:32:49.683372 31986 slave.cpp:6168] Container
'28f07a61-676a-4876-aae4-73598de90aae' for executor '1-0' of framework
80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
decode HTTP responses: Decoding failed
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 07 Nov 2018 08:32:46 GMT
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
105c
{"token":"eyJhbGciOiJSUzI1NiIsInR5cC...
Logs do not show the destination of the http request (a pull on docker
hub ? a request to master ? ...)
I could increase some slave logging, and HTTP failure occurs with
pulling of image:
I1107 10:45:56.689092 31987 registry_puller.cpp:286] Pulling image
'library/centos:latest' from
'docker-manifest://registry-1.docker.io:443library/centos?latest#https'
to '/tmp/mesos/store/docker/staging/99WUh3'
E1107 10:45:57.634601 31987 slave.cpp:6168] Container
'48ea5811-3f97-41c1-b1a5-9a4416552545' for executor '6-0' of framework
80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
decode HTTP responses: Decoding failed
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 07 Nov 2018 09:45:54 GMT
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
so it seems there is an issue with mesos unified management with docker
hub. Could it be related to libcurl version? (libcurl4 on my system). Is
specific setup needed?
any idea on what could be wrong ? or how to get more debug info ?
Thanks
Olivier
--
Olivier Sallou
Univ Rennes, Inria, CNRS, IRISA
Irisa, Campus de Beaulieu
F-35042 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org<http://keyring.debian.org>)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
--
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma
brièveté.
--
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma
brièveté.
Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
Hi Olivier and Adam,
This issue is already fixed by
https://issues.apache.org/jira/browse/MESOS-8907 which was merged and
backported 17 days ago. Please try it out and let me know.
More background contexts:
1. http-parse does not support http/2:
https://issues.apache.org/jira/browse/MESOS-8368
2. curl enable http/2 by default after curl 7.47
- Gilbert
On Thu, Nov 8, 2018 at 2:33 PM Cecile, Adam wrote:
> http-parser library is not updated anymore and does not support HTTP/2 so
> it's a dead end anyway
>
> Le 8 novembre 2018 10:17:20 GMT+01:00, Olivier Sallou <
> olivier.sal...@irisa.fr> a écrit :
>>
>>
>>
>> - Mail original -
>>
>>> De: "Olivier Sallou"
>>> À: user@mesos.apache.org
>>> Envoyé: Mercredi 7 Novembre 2018 12:40:24
>>> Objet: Re: mesos containerizer issue with v1.8.0: HTTP response Decoding
>>> failed
>>>
>>
>> On 11/7/18 11:54 AM, Cecile, Adam wrote:
>>>
>>>> Hi,
>>>>
>>>> You might be hitting the same bug as I did (no HTTP/2 support in code
>>>> pulling images for Mesos).
>>>> https://issues.apache.org/jira/browse/MESOS-9364
>>>>
>>>
>>> adding some logs as suggested in your issue, error code is different,
>>> getting "invalid constant string" error from http_parser error:
>>>
>>>
>>> XX(INVALID_CONSTANT, "invalid constant string")
>>>
>>
>>
>> with additional debug, problem is related to http parser vs received answer
>> from dockerhub. If I remove control on parsed data length vs body length it
>> works nicely (for test/debug only), so it may be a pb of response http
>> compliance...
>> Anyway, this issue prevents download of docker images when using unified
>> containerizer. As I am using code from master branch (latest), I can only
>> hope it will be fixed before next release
>>
>>
>>
>>
>>>
>>>
>>>> My report also include some code you can add in C++ code of the fetcher
>>>> to retrieve the actual message coming from the http response parser
>>>> library.
>>>>
>>>> Regards, Adam.
>>>>
>>>> On 11/7/18 11:42 AM, Adam Cecile wrote:
>>>>
>>>>> On 11/7/18 10:48 AM, Olivier Sallou wrote:
>>>>>
>>>>>> On 11/7/18 10:38 AM, Olivier Sallou wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I installed mesos from source. It works fine with docker containerizer.
>>>>>>>
>>>>>>> Howerver it fails with unified containerizer at container start.
>>>>>>>
>>>>>>> It used to work on a previous (older release) install. In the
>>>>>>> meanwhile,
>>>>>>> some system libs etc.. have been upgraded.
>>>>>>>
>>>>>>> In logs I have the following:
>>>>>>>
>>>>>>>
>>>>>>> I1107 09:32:48.707176 31983 containerizer.cpp:1280] Starting container
>>>>>>> 28f07a61-676a-4876-aae4-73598de90aae
>>>>>>> E1107 09:32:49.683372 31986 slave.cpp:6168] Container
>>>>>>> '28f07a61-676a-4876-aae4-73598de90aae' for executor '1-0' of framework
>>>>>>> 80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
>>>>>>> decode HTTP responses: Decoding failed
>>>>>>> HTTP/1.1 200 OK
>>>>>>> Content-Type: application/json
>>>>>>> Date: Wed, 07 Nov 2018 08:32:46 GMT
>>>>>>> Transfer-Encoding: chunked
>>>>>>> Strict-Transport-Security: max-age=31536000
>>>>>>>
>>>>>>> 105c
>>>>>>> {"token":"eyJhbGciOiJSUzI1NiIsInR5cC...
>>>>>>>
>>>>>>>
>>>>>>> Logs do not show the destination of the http request (a pull on docker
>>>>>>> hub ? a request to master ? ...)
>>>>>>>
>>>>>> I could increase some slave logging, and HTTP failure occurs with
>>>>>> pulling of image:
>>>>>>
>>>>>> I1107 10:45:56.689092 31987 registry_puller.cpp:286] Pulling image
>>>>>> 'library/centos:latest' from
>>>>>> 'docker-manifest://registry-1.docker.io:443library/centos?latest#https'
>>>>>> to '/tmp/mesos/store/docker/staging/99WUh3'
>>>>>> E1107 10:45:57.634601 31987 slave.cpp:6168] Container
>>>>>> '48ea5811-3f97-41c1-b1a5-9a4416552545' for executor '6-0' of framework
>>>>>> 80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
>>>>>> decode HTTP responses: Decoding failed
>>>>>> HTTP/1.1 200 OK
>>>>>> Content-Type: application/json
>>>>>> Date: Wed, 07 Nov 2018 09:45:54 GMT
>>>>>> Transfer-Encoding: chunked
>>>>>> Strict-Transport-Security: max-age=31536000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> so it seems there is an issue with mesos unified management with docker
>>>>>> hub. Could it be related to libcurl version? (libcurl4 on my system). Is
>>>>>> specific setup needed?
>>>>>>
>>>>>>
>>>>>>
>>>>>> any idea on what could be wrong ? or how to get more debug info ?
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>> Olivier
>>>>>>>
>>>>>>>
>>>>>>> --
>>> Olivier Sallou
>>> Univ Rennes, Inria, CNRS, IRISA
>>> Irisa, Campus de Beaulieu
>>> F-35042 RENNES - FRANCE
>>> Tel: 02.99.84.71.95
>>>
>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>>
>>
> --
> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma
> brièveté.
>
Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
Olivier, are you sure you're talking to dockerhub directly? Is there a
proxy in the middle?
I think it's also likely that it's related to your curl version.
I tested on today's master branch, using this marathon config (docker image
is centos). It works for me.
{
"id": "/test",
"instances": 1,
"portDefinitions": [],
"container": {
"type": "MESOS",
"volumes": [],
"docker": {
"image": "centos"
}
},
"cpus": 0.1,
"mem": 128,
"requirePorts": false,
"networks": [],
"healthChecks": [],
"fetch": [],
"constraints": [],
"cmd": "sleep 1"
}
- Jie
On Thu, Nov 8, 2018 at 2:33 PM Cecile, Adam wrote:
> http-parser library is not updated anymore and does not support HTTP/2 so
> it's a dead end anyway
>
> Le 8 novembre 2018 10:17:20 GMT+01:00, Olivier Sallou <
> olivier.sal...@irisa.fr> a écrit :
>>
>>
>>
>> - Mail original -
>>
>>> De: "Olivier Sallou"
>>> À: user@mesos.apache.org
>>> Envoyé: Mercredi 7 Novembre 2018 12:40:24
>>> Objet: Re: mesos containerizer issue with v1.8.0: HTTP response Decoding
>>> failed
>>>
>>
>> On 11/7/18 11:54 AM, Cecile, Adam wrote:
>>>
>>>> Hi,
>>>>
>>>> You might be hitting the same bug as I did (no HTTP/2 support in code
>>>> pulling images for Mesos).
>>>> https://issues.apache.org/jira/browse/MESOS-9364
>>>>
>>>
>>> adding some logs as suggested in your issue, error code is different,
>>> getting "invalid constant string" error from http_parser error:
>>>
>>>
>>> XX(INVALID_CONSTANT, "invalid constant string")
>>>
>>
>>
>> with additional debug, problem is related to http parser vs received answer
>> from dockerhub. If I remove control on parsed data length vs body length it
>> works nicely (for test/debug only), so it may be a pb of response http
>> compliance...
>> Anyway, this issue prevents download of docker images when using unified
>> containerizer. As I am using code from master branch (latest), I can only
>> hope it will be fixed before next release
>>
>>
>>
>>
>>>
>>>
>>>> My report also include some code you can add in C++ code of the fetcher
>>>> to retrieve the actual message coming from the http response parser
>>>> library.
>>>>
>>>> Regards, Adam.
>>>>
>>>> On 11/7/18 11:42 AM, Adam Cecile wrote:
>>>>
>>>>> On 11/7/18 10:48 AM, Olivier Sallou wrote:
>>>>>
>>>>>> On 11/7/18 10:38 AM, Olivier Sallou wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I installed mesos from source. It works fine with docker containerizer.
>>>>>>>
>>>>>>> Howerver it fails with unified containerizer at container start.
>>>>>>>
>>>>>>> It used to work on a previous (older release) install. In the
>>>>>>> meanwhile,
>>>>>>> some system libs etc.. have been upgraded.
>>>>>>>
>>>>>>> In logs I have the following:
>>>>>>>
>>>>>>>
>>>>>>> I1107 09:32:48.707176 31983 containerizer.cpp:1280] Starting container
>>>>>>> 28f07a61-676a-4876-aae4-73598de90aae
>>>>>>> E1107 09:32:49.683372 31986 slave.cpp:6168] Container
>>>>>>> '28f07a61-676a-4876-aae4-73598de90aae' for executor '1-0' of framework
>>>>>>> 80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
>>>>>>> decode HTTP responses: Decoding failed
>>>>>>> HTTP/1.1 200 OK
>>>>>>> Content-Type: application/json
>>>>>>> Date: Wed, 07 Nov 2018 08:32:46 GMT
>>>>>>> Transfer-Encoding: chunked
>>>>>>> Strict-Transport-Security: max-age=31536000
>>>>>>>
>>>>>>> 105c
>>>>>>> {"token":"eyJhbGciOiJSUzI1NiIsInR5cC...
>>>>>>>
>>>>>>>
>>>>>>> Logs do not show the destination of the http request (a pull on docker
>>>>>>> hub ? a request to master
Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
http-parser library is not updated anymore and does not support HTTP/2 so it's
a dead end anyway
Le 8 novembre 2018 10:17:20 GMT+01:00, Olivier Sallou
a écrit :
- Mail original -
De: "Olivier Sallou"
À: user@mesos.apache.org
Envoyé: Mercredi 7 Novembre 2018 12:40:24
Objet: Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
On 11/7/18 11:54 AM, Cecile, Adam wrote:
Hi,
You might be hitting the same bug as I did (no HTTP/2 support in code
pulling images for Mesos).
https://issues.apache.org/jira/browse/MESOS-9364
adding some logs as suggested in your issue, error code is different,
getting "invalid constant string" error from http_parser error:
XX(INVALID_CONSTANT, "invalid constant string")
with additional debug, problem is related to http parser vs received answer
from dockerhub. If I remove control on parsed data length vs body length it
works nicely (for test/debug only), so it may be a pb of response http
compliance...
Anyway, this issue prevents download of docker images when using unified
containerizer. As I am using code from master branch (latest), I can only hope
it will be fixed before next release
My report also include some code you can add in C++ code of the fetcher
to retrieve the actual message coming from the http response parser library.
Regards, Adam.
On 11/7/18 11:42 AM, Adam Cecile wrote:
On 11/7/18 10:48 AM, Olivier Sallou wrote:
On 11/7/18 10:38 AM, Olivier Sallou wrote:
Hi,
I installed mesos from source. It works fine with docker containerizer.
Howerver it fails with unified containerizer at container start.
It used to work on a previous (older release) install. In the
meanwhile,
some system libs etc.. have been upgraded.
In logs I have the following:
I1107 09:32:48.707176 31983 containerizer.cpp:1280] Starting container
28f07a61-676a-4876-aae4-73598de90aae
E1107 09:32:49.683372 31986 slave.cpp:6168] Container
'28f07a61-676a-4876-aae4-73598de90aae' for executor '1-0' of framework
80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
decode HTTP responses: Decoding failed
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 07 Nov 2018 08:32:46 GMT
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
105c
{"token":"eyJhbGciOiJSUzI1NiIsInR5cC...
Logs do not show the destination of the http request (a pull on docker
hub ? a request to master ? ...)
I could increase some slave logging, and HTTP failure occurs with
pulling of image:
I1107 10:45:56.689092 31987 registry_puller.cpp:286] Pulling image
'library/centos:latest' from
'docker-manifest://registry-1.docker.io:443library/centos?latest#https'
to '/tmp/mesos/store/docker/staging/99WUh3'
E1107 10:45:57.634601 31987 slave.cpp:6168] Container
'48ea5811-3f97-41c1-b1a5-9a4416552545' for executor '6-0' of framework
80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
decode HTTP responses: Decoding failed
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 07 Nov 2018 09:45:54 GMT
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
so it seems there is an issue with mesos unified management with docker
hub. Could it be related to libcurl version? (libcurl4 on my system). Is
specific setup needed?
any idea on what could be wrong ? or how to get more debug info ?
Thanks
Olivier
--
Olivier Sallou
Univ Rennes, Inria, CNRS, IRISA
Irisa, Campus de Beaulieu
F-35042 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
--
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma
brièveté.
Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
Hi,
Actually I'm wrapping docker client pull request and my server was
running HTTP/2 only.
Check http-parser you may find a hint regarding this "bad" constant.
Regards,
On 11/7/18 12:40 PM, Olivier Sallou wrote:
> On 11/7/18 11:54 AM, Cecile, Adam wrote:
>> Hi,
>>
>> You might be hitting the same bug as I did (no HTTP/2 support in code
>> pulling images for Mesos).
>> https://issues.apache.org/jira/browse/MESOS-9364
> adding some logs as suggested in your issue, error code is different,
> getting "invalid constant string" error from http_parser error:
>
>
> XX(INVALID_CONSTANT, "invalid constant string")
>
>
>> My report also include some code you can add in C++ code of the fetcher
>> to retrieve the actual message coming from the http response parser library.
>>
>> Regards, Adam.
>>
>> On 11/7/18 11:42 AM, Adam Cecile wrote:
>>> On 11/7/18 10:48 AM, Olivier Sallou wrote:
On 11/7/18 10:38 AM, Olivier Sallou wrote:
> Hi,
>
> I installed mesos from source. It works fine with docker containerizer.
>
> Howerver it fails with unified containerizer at container start.
>
> It used to work on a previous (older release) install. In the
> meanwhile,
> some system libs etc.. have been upgraded.
>
> In logs I have the following:
>
>
> I1107 09:32:48.707176 31983 containerizer.cpp:1280] Starting container
> 28f07a61-676a-4876-aae4-73598de90aae
> E1107 09:32:49.683372 31986 slave.cpp:6168] Container
> '28f07a61-676a-4876-aae4-73598de90aae' for executor '1-0' of framework
> 80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
> decode HTTP responses: Decoding failed
> HTTP/1.1 200 OK
> Content-Type: application/json
> Date: Wed, 07 Nov 2018 08:32:46 GMT
> Transfer-Encoding: chunked
> Strict-Transport-Security: max-age=31536000
>
> 105c
> {"token":"eyJhbGciOiJSUzI1NiIsInR5cC...
>
>
> Logs do not show the destination of the http request (a pull on docker
> hub ? a request to master ? ...)
I could increase some slave logging, and HTTP failure occurs with
pulling of image:
I1107 10:45:56.689092 31987 registry_puller.cpp:286] Pulling image
'library/centos:latest' from
'docker-manifest://registry-1.docker.io:443library/centos?latest#https'
to '/tmp/mesos/store/docker/staging/99WUh3'
E1107 10:45:57.634601 31987 slave.cpp:6168] Container
'48ea5811-3f97-41c1-b1a5-9a4416552545' for executor '6-0' of framework
80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
decode HTTP responses: Decoding failed
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 07 Nov 2018 09:45:54 GMT
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
so it seems there is an issue with mesos unified management with docker
hub. Could it be related to libcurl version? (libcurl4 on my system). Is
specific setup needed?
> any idea on what could be wrong ? or how to get more debug info ?
>
>
> Thanks
>
>
> Olivier
>
>
Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
On 11/7/18 11:54 AM, Cecile, Adam wrote:
> Hi,
>
> You might be hitting the same bug as I did (no HTTP/2 support in code
> pulling images for Mesos).
> https://issues.apache.org/jira/browse/MESOS-9364
adding some logs as suggested in your issue, error code is different,
getting "invalid constant string" error from http_parser error:
XX(INVALID_CONSTANT, "invalid constant string")
>
> My report also include some code you can add in C++ code of the fetcher
> to retrieve the actual message coming from the http response parser library.
>
> Regards, Adam.
>
> On 11/7/18 11:42 AM, Adam Cecile wrote:
>> On 11/7/18 10:48 AM, Olivier Sallou wrote:
>>> On 11/7/18 10:38 AM, Olivier Sallou wrote:
Hi,
I installed mesos from source. It works fine with docker containerizer.
Howerver it fails with unified containerizer at container start.
It used to work on a previous (older release) install. In the
meanwhile,
some system libs etc.. have been upgraded.
In logs I have the following:
I1107 09:32:48.707176 31983 containerizer.cpp:1280] Starting container
28f07a61-676a-4876-aae4-73598de90aae
E1107 09:32:49.683372 31986 slave.cpp:6168] Container
'28f07a61-676a-4876-aae4-73598de90aae' for executor '1-0' of framework
80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
decode HTTP responses: Decoding failed
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 07 Nov 2018 08:32:46 GMT
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
105c
{"token":"eyJhbGciOiJSUzI1NiIsInR5cC...
Logs do not show the destination of the http request (a pull on docker
hub ? a request to master ? ...)
>>> I could increase some slave logging, and HTTP failure occurs with
>>> pulling of image:
>>>
>>> I1107 10:45:56.689092 31987 registry_puller.cpp:286] Pulling image
>>> 'library/centos:latest' from
>>> 'docker-manifest://registry-1.docker.io:443library/centos?latest#https'
>>> to '/tmp/mesos/store/docker/staging/99WUh3'
>>> E1107 10:45:57.634601 31987 slave.cpp:6168] Container
>>> '48ea5811-3f97-41c1-b1a5-9a4416552545' for executor '6-0' of framework
>>> 80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
>>> decode HTTP responses: Decoding failed
>>> HTTP/1.1 200 OK
>>> Content-Type: application/json
>>> Date: Wed, 07 Nov 2018 09:45:54 GMT
>>> Transfer-Encoding: chunked
>>> Strict-Transport-Security: max-age=31536000
>>>
>>>
>>>
>>>
>>> so it seems there is an issue with mesos unified management with docker
>>> hub. Could it be related to libcurl version? (libcurl4 on my system). Is
>>> specific setup needed?
>>>
>>>
>>>
any idea on what could be wrong ? or how to get more debug info ?
Thanks
Olivier
--
Olivier Sallou
Univ Rennes, Inria, CNRS, IRISA
Irisa, Campus de Beaulieu
F-35042 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
On 11/7/18 11:54 AM, Cecile, Adam wrote:
> Hi,
>
> You might be hitting the same bug as I did (no HTTP/2 support in code
> pulling images for Mesos).
> https://issues.apache.org/jira/browse/MESOS-9364
>
> My report also include some code you can add in C++ code of the fetcher
> to retrieve the actual message coming from the http response parser library.
could be indeed. thanks for pointing to it.
I see however that it fails for you with mesos 1.4.1 ? I tested mesos
mode image pulling some time ago with mesos 1.7 and it worked.
Is it a modification on dockerhub using http/2 only ?
>
> Regards, Adam.
>
> On 11/7/18 11:42 AM, Adam Cecile wrote:
>> On 11/7/18 10:48 AM, Olivier Sallou wrote:
>>> On 11/7/18 10:38 AM, Olivier Sallou wrote:
Hi,
I installed mesos from source. It works fine with docker containerizer.
Howerver it fails with unified containerizer at container start.
It used to work on a previous (older release) install. In the
meanwhile,
some system libs etc.. have been upgraded.
In logs I have the following:
I1107 09:32:48.707176 31983 containerizer.cpp:1280] Starting container
28f07a61-676a-4876-aae4-73598de90aae
E1107 09:32:49.683372 31986 slave.cpp:6168] Container
'28f07a61-676a-4876-aae4-73598de90aae' for executor '1-0' of framework
80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
decode HTTP responses: Decoding failed
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 07 Nov 2018 08:32:46 GMT
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
105c
{"token":"eyJhbGciOiJSUzI1NiIsInR5cC...
Logs do not show the destination of the http request (a pull on docker
hub ? a request to master ? ...)
>>> I could increase some slave logging, and HTTP failure occurs with
>>> pulling of image:
>>>
>>> I1107 10:45:56.689092 31987 registry_puller.cpp:286] Pulling image
>>> 'library/centos:latest' from
>>> 'docker-manifest://registry-1.docker.io:443library/centos?latest#https'
>>> to '/tmp/mesos/store/docker/staging/99WUh3'
>>> E1107 10:45:57.634601 31987 slave.cpp:6168] Container
>>> '48ea5811-3f97-41c1-b1a5-9a4416552545' for executor '6-0' of framework
>>> 80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
>>> decode HTTP responses: Decoding failed
>>> HTTP/1.1 200 OK
>>> Content-Type: application/json
>>> Date: Wed, 07 Nov 2018 09:45:54 GMT
>>> Transfer-Encoding: chunked
>>> Strict-Transport-Security: max-age=31536000
>>>
>>>
>>>
>>>
>>> so it seems there is an issue with mesos unified management with docker
>>> hub. Could it be related to libcurl version? (libcurl4 on my system). Is
>>> specific setup needed?
>>>
>>>
>>>
any idea on what could be wrong ? or how to get more debug info ?
Thanks
Olivier
--
Olivier Sallou
Univ Rennes, Inria, CNRS, IRISA
Irisa, Campus de Beaulieu
F-35042 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
Hi,
You might be hitting the same bug as I did (no HTTP/2 support in code
pulling images for Mesos).
https://issues.apache.org/jira/browse/MESOS-9364
My report also include some code you can add in C++ code of the fetcher
to retrieve the actual message coming from the http response parser library.
Regards, Adam.
On 11/7/18 11:42 AM, Adam Cecile wrote:
> On 11/7/18 10:48 AM, Olivier Sallou wrote:
>> On 11/7/18 10:38 AM, Olivier Sallou wrote:
>>> Hi,
>>>
>>> I installed mesos from source. It works fine with docker containerizer.
>>>
>>> Howerver it fails with unified containerizer at container start.
>>>
>>> It used to work on a previous (older release) install. In the
>>> meanwhile,
>>> some system libs etc.. have been upgraded.
>>>
>>> In logs I have the following:
>>>
>>>
>>> I1107 09:32:48.707176 31983 containerizer.cpp:1280] Starting container
>>> 28f07a61-676a-4876-aae4-73598de90aae
>>> E1107 09:32:49.683372 31986 slave.cpp:6168] Container
>>> '28f07a61-676a-4876-aae4-73598de90aae' for executor '1-0' of framework
>>> 80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
>>> decode HTTP responses: Decoding failed
>>> HTTP/1.1 200 OK
>>> Content-Type: application/json
>>> Date: Wed, 07 Nov 2018 08:32:46 GMT
>>> Transfer-Encoding: chunked
>>> Strict-Transport-Security: max-age=31536000
>>>
>>> 105c
>>> {"token":"eyJhbGciOiJSUzI1NiIsInR5cC...
>>>
>>>
>>> Logs do not show the destination of the http request (a pull on docker
>>> hub ? a request to master ? ...)
>> I could increase some slave logging, and HTTP failure occurs with
>> pulling of image:
>>
>> I1107 10:45:56.689092 31987 registry_puller.cpp:286] Pulling image
>> 'library/centos:latest' from
>> 'docker-manifest://registry-1.docker.io:443library/centos?latest#https'
>> to '/tmp/mesos/store/docker/staging/99WUh3'
>> E1107 10:45:57.634601 31987 slave.cpp:6168] Container
>> '48ea5811-3f97-41c1-b1a5-9a4416552545' for executor '6-0' of framework
>> 80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
>> decode HTTP responses: Decoding failed
>> HTTP/1.1 200 OK
>> Content-Type: application/json
>> Date: Wed, 07 Nov 2018 09:45:54 GMT
>> Transfer-Encoding: chunked
>> Strict-Transport-Security: max-age=31536000
>>
>>
>>
>>
>> so it seems there is an issue with mesos unified management with docker
>> hub. Could it be related to libcurl version? (libcurl4 on my system). Is
>> specific setup needed?
>>
>>
>>
>>>
>>> any idea on what could be wrong ? or how to get more debug info ?
>>>
>>>
>>> Thanks
>>>
>>>
>>> Olivier
>>>
>>>
>
Re: mesos containerizer issue with v1.8.0: HTTP response Decoding failed
On 11/7/18 10:38 AM, Olivier Sallou wrote:
> Hi,
>
> I installed mesos from source. It works fine with docker containerizer.
>
> Howerver it fails with unified containerizer at container start.
>
> It used to work on a previous (older release) install. In the meanwhile,
> some system libs etc.. have been upgraded.
>
> In logs I have the following:
>
>
> I1107 09:32:48.707176 31983 containerizer.cpp:1280] Starting container
> 28f07a61-676a-4876-aae4-73598de90aae
> E1107 09:32:49.683372 31986 slave.cpp:6168] Container
> '28f07a61-676a-4876-aae4-73598de90aae' for executor '1-0' of framework
> 80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
> decode HTTP responses: Decoding failed
> HTTP/1.1 200 OK
> Content-Type: application/json
> Date: Wed, 07 Nov 2018 08:32:46 GMT
> Transfer-Encoding: chunked
> Strict-Transport-Security: max-age=31536000
>
> 105c
> {"token":"eyJhbGciOiJSUzI1NiIsInR5cC...
>
>
> Logs do not show the destination of the http request (a pull on docker
> hub ? a request to master ? ...)
I could increase some slave logging, and HTTP failure occurs with
pulling of image:
I1107 10:45:56.689092 31987 registry_puller.cpp:286] Pulling image
'library/centos:latest' from
'docker-manifest://registry-1.docker.io:443library/centos?latest#https'
to '/tmp/mesos/store/docker/staging/99WUh3'
E1107 10:45:57.634601 31987 slave.cpp:6168] Container
'48ea5811-3f97-41c1-b1a5-9a4416552545' for executor '6-0' of framework
80fc2079-ba14-454b-8276-79fae090f8b3- failed to start: Failed to
decode HTTP responses: Decoding failed
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 07 Nov 2018 09:45:54 GMT
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000
so it seems there is an issue with mesos unified management with docker
hub. Could it be related to libcurl version? (libcurl4 on my system). Is
specific setup needed?
>
>
> any idea on what could be wrong ? or how to get more debug info ?
>
>
> Thanks
>
>
> Olivier
>
>
--
Olivier Sallou
Univ Rennes, Inria, CNRS, IRISA
Irisa, Campus de Beaulieu
F-35042 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
