[jira] [Comment Edited] (MESOS-6010) Docker registry puller shows decode error "No response decoded".
[
https://issues.apache.org/jira/browse/MESOS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15822158#comment-15822158
]
Jie Yu edited comment on MESOS-6010 at 1/13/17 7:11 PM:
Did some digging today on this issue.
1) As [~nfnt] mentioned, there is a proxy between agent and docker registry
(e.g., squid) that is doing HTTP CONNECT tunneling (e.g.,
http://wiki.squid-cache.org/Features/HTTPS).
2) Recent versions of curl (after 7.11.1) starts to include the proxy response
(200 Connection established) as well in the response
(https://curl.haxx.se/mail/lib-2005-10/0024.html).
3) According to RFC, it's ok to have zero headers (See 4.1, "zero or more
header fields", https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.1).
4) According to RFC, if no header is specified, client should read the response
body till EOF (See 4.4,
https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4).
5) I think our http parser does the right thing, since we didn't feed the
decoder an EOF (i.e., decoder->decode("", 0)), the parser thinks that there
will be more body content from the socket.
6) That results in "No response decoded".
Some relevant thread:
https://github.com/nodejs/node-v0.x-archive/issues/1711
https://github.com/nodejs/http-parser/issues/327
https://github.com/nodejs/node-v0.x-archive/issues/1956
https://curl.haxx.se/mail/lib-2005-10/0023.html
http://stackoverflow.com/questions/16965530/what-to-do-with-extra-http-header-from-proxy
was (Author: jieyu):
Did some digging today on this issue.
1) As [~nfnt] mentioned, there is a proxy between agent and docker registry
(e.g., squid) that is doing HTTP CONNECT tunneling (e.g.,
http://wiki.squid-cache.org/Features/HTTPS).
2) Recent versions of curl (after 7.11.1) starts to include the proxy response
(200 Connection established) as well in the response
(https://curl.haxx.se/mail/lib-2005-10/0024.html).
3) According to RFC, it's ok to have zero headers (See 4.1, "zero or more
header fields", https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.1).
4) According to RFC, if no header is specified, client should read the response
body till EOF (See 4.4,
https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4).
5) I think our http parser does the right thing, since we didn't feed the
decoder an EOF (i.e., decoder->decode("", 0)), the parser thinks that there
will be more body content from the socket.
6) That results in "No response decoded".
Some relevant thread:
https://github.com/nodejs/node-v0.x-archive/issues/1711
https://github.com/nodejs/http-parser/issues/327
https://github.com/nodejs/node-v0.x-archive/issues/1956
https://curl.haxx.se/mail/lib-2005-10/0023.html
> Docker registry puller shows decode error "No response decoded".
>
>
> Key: MESOS-6010
> URL: https://issues.apache.org/jira/browse/MESOS-6010
> Project: Mesos
> Issue Type: Bug
> Components: containerization, docker
>Affects Versions: 1.0.0, 1.0.1
>Reporter: Sunzhe
>Assignee: Jan Schlicht
>Priority: Critical
> Labels: Docker, mesos-containerizer
>
> The {{mesos-agent}} flags:
> {code}
> GLOG_v=1 ./bin/mesos-agent.sh \
> --master=zk://${MESOS_MASTER_IP}:2181/mesos \
> --ip=10.100.3.3 \
> --work_dir=${MESOS_WORK_DIR} \
>
> --isolation=cgroups/devices,gpu/nvidia,disk/du,docker/runtime,filesystem/linux
> \
> --enforce_container_disk_quota \
> --containerizers=mesos \
> --image_providers=docker \
> --executor_environment_variables="{}"
> {code}
> And the {{mesos-execute}} flags:
> {code}
> ./src/mesos-execute \
>--master=${MESOS_MASTER_IP}:5050 \
>--name=${INSTANCE_NAME} \
>--docker_image=${DOCKER_IMAGE} \
>--framework_capabilities=GPU_RESOURCES \
>--shell=false
> {code}
> But when {{./src/mesos-execute}}, the errors like below:
> {code}
> I0809 16:11:46.207875 25583 scheduler.cpp:172] Version: 1.0.0
> I0809 16:11:46.212442 25582 scheduler.cpp:461] New master detected at
> master@10.103.0.125:5050
> Subscribed with ID '168ab900-ee7e-4829-a59a-d16de956637e-0009'
> Submitted task 'test' to agent '168ab900-ee7e-4829-a59a-d16de956637e-S1'
> Received status update TASK_FAILED for task 'test'
> message: 'Failed to launch container: Failed to decode HTTP responses: No
> response decoded
> HTTP/1.1 200 Connection established
> HTTP/1.1 401 Unauthorized
> Content-Type: application/json; charset=utf-8
> Docker-Distribution-Api-Version: registry/2.0
> Www-Authenticate: Bearer
> realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/redis:pull";
> Date: Tue, 09 Aug 2016 08:10:32
[jira] [Comment Edited] (MESOS-6010) Docker registry puller shows decode error "No response decoded".
[
https://issues.apache.org/jira/browse/MESOS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15822158#comment-15822158
]
Jie Yu edited comment on MESOS-6010 at 1/13/17 7:07 PM:
Did some digging today on this issue.
1) As [~nfnt] mentioned, there is a proxy between agent and docker registry
(e.g., squid) that is doing HTTP CONNECT tunneling (e.g.,
http://wiki.squid-cache.org/Features/HTTPS).
2) Recent versions of curl (after 7.11.1) starts to include the proxy response
(200 Connection established) as well in the response
(https://curl.haxx.se/mail/lib-2005-10/0024.html).
3) According to RFC, it's ok to have zero headers (See 4.1, "zero or more
header fields", https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.1).
4) According to RFC, if no header is specified, client should read the response
body till EOF (See 4.4,
https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4).
5) I think our http parser does the right thing, since we didn't feed the
decoder an EOF (i.e., decoder->decode("", 0)), the parser thinks that there
will be more body content from the socket.
6) That results in "No response decoded".
Some relevant thread:
https://github.com/nodejs/node-v0.x-archive/issues/1711
https://github.com/nodejs/http-parser/issues/327
https://github.com/nodejs/node-v0.x-archive/issues/1956
https://curl.haxx.se/mail/lib-2005-10/0023.html
was (Author: jieyu):
Did some digging today on this issue.
1) As [~nfnt] mentioned, there is a proxy between agent and docker registry
(e.g., squid) that is doing HTTP CONNECT tunneling (e.g.,
http://wiki.squid-cache.org/Features/HTTPS).
2) Recent versions of curl (after 7.11.1) starts to include the proxy response
(200 Connection established) as well in the response
(https://curl.haxx.se/mail/lib-2005-10/0024.html).
3) According to RFC, it's ok to have zero headers (See 4.1, "zero or more
header fields", https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.1).
4) According to RFC, if no header is specified, client should read the response
body till EOF (See 4.4,
https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4).
5) I think our http parser does the right thing, since we didn't feed the
decoder an EOF (i.e., decoder->decode("", 0)), the parser thinks that there
will be more body content from the socket.
6) That results in "No response decoded".
Some relevant thread:
https://github.com/nodejs/node-v0.x-archive/issues/1711
https://github.com/nodejs/http-parser/issues/327
https://github.com/nodejs/node-v0.x-archive/issues/1956
> Docker registry puller shows decode error "No response decoded".
>
>
> Key: MESOS-6010
> URL: https://issues.apache.org/jira/browse/MESOS-6010
> Project: Mesos
> Issue Type: Bug
> Components: containerization, docker
>Affects Versions: 1.0.0, 1.0.1
>Reporter: Sunzhe
>Assignee: Jan Schlicht
>Priority: Critical
> Labels: Docker, mesos-containerizer
>
> The {{mesos-agent}} flags:
> {code}
> GLOG_v=1 ./bin/mesos-agent.sh \
> --master=zk://${MESOS_MASTER_IP}:2181/mesos \
> --ip=10.100.3.3 \
> --work_dir=${MESOS_WORK_DIR} \
>
> --isolation=cgroups/devices,gpu/nvidia,disk/du,docker/runtime,filesystem/linux
> \
> --enforce_container_disk_quota \
> --containerizers=mesos \
> --image_providers=docker \
> --executor_environment_variables="{}"
> {code}
> And the {{mesos-execute}} flags:
> {code}
> ./src/mesos-execute \
>--master=${MESOS_MASTER_IP}:5050 \
>--name=${INSTANCE_NAME} \
>--docker_image=${DOCKER_IMAGE} \
>--framework_capabilities=GPU_RESOURCES \
>--shell=false
> {code}
> But when {{./src/mesos-execute}}, the errors like below:
> {code}
> I0809 16:11:46.207875 25583 scheduler.cpp:172] Version: 1.0.0
> I0809 16:11:46.212442 25582 scheduler.cpp:461] New master detected at
> master@10.103.0.125:5050
> Subscribed with ID '168ab900-ee7e-4829-a59a-d16de956637e-0009'
> Submitted task 'test' to agent '168ab900-ee7e-4829-a59a-d16de956637e-S1'
> Received status update TASK_FAILED for task 'test'
> message: 'Failed to launch container: Failed to decode HTTP responses: No
> response decoded
> HTTP/1.1 200 Connection established
> HTTP/1.1 401 Unauthorized
> Content-Type: application/json; charset=utf-8
> Docker-Distribution-Api-Version: registry/2.0
> Www-Authenticate: Bearer
> realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/redis:pull";
> Date: Tue, 09 Aug 2016 08:10:32 GMT
> Content-Length: 145
> Strict-Transport-Security: max-age=31536000
> {"errors":[{"code":"UNAUTHORIZED","message":"authentication
>
[jira] [Comment Edited] (MESOS-6010) Docker registry puller shows decode error "No response decoded".
[
https://issues.apache.org/jira/browse/MESOS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15822158#comment-15822158
]
Jie Yu edited comment on MESOS-6010 at 1/13/17 6:57 PM:
Did some digging today on this issue.
1) As [~nfnt] mentioned, there is a proxy between agent and docker registry
(e.g., squid) that is doing HTTP CONNECT tunneling (e.g.,
http://wiki.squid-cache.org/Features/HTTPS).
2) Recent versions of curl (after 7.11.1) starts to include the proxy response
(200 Connection established) as well in the response
(https://curl.haxx.se/mail/lib-2005-10/0024.html).
3) According to RFC, it's ok to have zero headers (See 4.1, "zero or more
header fields", https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.1).
4) According to RFC, if no header is specified, client should read the response
body till EOF (See 4.4,
https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4).
5) I think our http parser does the right thing, since we didn't feed the
decoder an EOF (i.e., decoder->decode("", 0)), the parser thinks that there
will be more body content from the socket.
6) That results in "No response decoded".
Some relevant thread:
https://github.com/nodejs/node-v0.x-archive/issues/1711
https://github.com/nodejs/http-parser/issues/327
https://github.com/nodejs/node-v0.x-archive/issues/1956
was (Author: jieyu):
Did some digging today on this issue.
1) As [~nfnt] mentioned, there is a proxy between agent and docker registry
(e.g., squid) that is doing HTTP CONNECT tunneling (e.g.,
http://wiki.squid-cache.org/Features/HTTPS).
2) Recent versions of curl (after 7.11.1) starts to include the proxy response
(200 Connection established) as well in the response
(https://curl.haxx.se/mail/lib-2005-10/0024.html).
3) According to RFC, it's ok to have zero headers (See 4.1, "zero or more
header fields", https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.1).
4) According to RFC, if no header is specified, client should read the response
body till EOF (See 4.4,
https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4).
5) I think out http parser does the right thing, since we didn't feed the
decoder an EOF (i.e., decoder->decode("", 0)), the parser thinks that there
will be more body content from the socket.
6) That results in "No response decoded".
Some relevant thread:
https://github.com/nodejs/node-v0.x-archive/issues/1711
https://github.com/nodejs/http-parser/issues/327
https://github.com/nodejs/node-v0.x-archive/issues/1956
> Docker registry puller shows decode error "No response decoded".
>
>
> Key: MESOS-6010
> URL: https://issues.apache.org/jira/browse/MESOS-6010
> Project: Mesos
> Issue Type: Bug
> Components: containerization, docker
>Affects Versions: 1.0.0, 1.0.1
>Reporter: Sunzhe
>Assignee: Jan Schlicht
>Priority: Critical
> Labels: Docker, mesos-containerizer
>
> The {{mesos-agent}} flags:
> {code}
> GLOG_v=1 ./bin/mesos-agent.sh \
> --master=zk://${MESOS_MASTER_IP}:2181/mesos \
> --ip=10.100.3.3 \
> --work_dir=${MESOS_WORK_DIR} \
>
> --isolation=cgroups/devices,gpu/nvidia,disk/du,docker/runtime,filesystem/linux
> \
> --enforce_container_disk_quota \
> --containerizers=mesos \
> --image_providers=docker \
> --executor_environment_variables="{}"
> {code}
> And the {{mesos-execute}} flags:
> {code}
> ./src/mesos-execute \
>--master=${MESOS_MASTER_IP}:5050 \
>--name=${INSTANCE_NAME} \
>--docker_image=${DOCKER_IMAGE} \
>--framework_capabilities=GPU_RESOURCES \
>--shell=false
> {code}
> But when {{./src/mesos-execute}}, the errors like below:
> {code}
> I0809 16:11:46.207875 25583 scheduler.cpp:172] Version: 1.0.0
> I0809 16:11:46.212442 25582 scheduler.cpp:461] New master detected at
> master@10.103.0.125:5050
> Subscribed with ID '168ab900-ee7e-4829-a59a-d16de956637e-0009'
> Submitted task 'test' to agent '168ab900-ee7e-4829-a59a-d16de956637e-S1'
> Received status update TASK_FAILED for task 'test'
> message: 'Failed to launch container: Failed to decode HTTP responses: No
> response decoded
> HTTP/1.1 200 Connection established
> HTTP/1.1 401 Unauthorized
> Content-Type: application/json; charset=utf-8
> Docker-Distribution-Api-Version: registry/2.0
> Www-Authenticate: Bearer
> realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/redis:pull";
> Date: Tue, 09 Aug 2016 08:10:32 GMT
> Content-Length: 145
> Strict-Transport-Security: max-age=31536000
> {"errors":[{"code":"UNAUTHORIZED","message":"authentication
> required","detail":[{"Type":"repository","Name"
[jira] [Comment Edited] (MESOS-6010) Docker registry puller shows decode error "No response decoded".
[
https://issues.apache.org/jira/browse/MESOS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15820967#comment-15820967
]
Jan Schlicht edited comment on MESOS-6010 at 1/12/17 1:17 PM:
--
To reproduce what's going on here in detail:
{noformat}
docker run -d -p 3128:3128 minimum2scp/squid
export https_proxy=127.0.0.1:3128
curl -vvv https://user:p...@httpbin.org/basic-auth/user/pass
{noformat}
which will produce this output:
{noformat}
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
* Establish HTTP proxy tunnel to httpbin.org:443
* Server auth using Basic with user 'user'
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.51.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: *.httpbin.org
* Server certificate: COMODO RSA Domain Validation Secure Server CA
* Server certificate: COMODO RSA Certification Authority
* Server auth using Basic with user 'user'
> GET /basic-auth/user/pass HTTP/1.1
> Host: httpbin.org
> Authorization: Basic dXNlcjpwYXNz
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 12 Jan 2017 13:05:00 GMT
< Content-Type: application/json
< Content-Length: 47
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
<
{
"authenticated": true,
"user": "user"
}
* Curl_http_done: called premature == 0
* Connection #0 to host 127.0.0.1 left intact
{noformat}
To use SSL through a proxy, curl uses [HTTP CONNECT
tunnelling|https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_tunneling].
This means that 2 HTTP requests are made and we get 2 HTTP responses. The first
response is a problem for {{http_parser}} because it doesn't have any headers.
We can force {{http_parser}} to parse this anyways but then the body of that
message would be the second HTTP response.
was (Author: nfnt):
To reproduce what's going on here in detail:
{noformat}
docker run -d -p 3128:3128 minimum2scp/squid
export https_proxy=127.0.0.1:3128
curl -vvv https://user:p...@httpbin.org/basic-auth/user/pass
{noformat}
To use SSL through a proxy, curl uses [HTTP CONNECT
tunnelling|https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_tunneling].
This means that 2 HTTP requests are made and we get 2 HTTP responses. The first
response is a problem for {{http_parser}} because it doesn't have any headers.
We can force {{http_parser}} to parse this anyways but then the body of that
message would be the second HTTP response.
> Docker registry puller shows decode error "No response decoded".
>
>
> Key: MESOS-6010
> URL: https://issues.apache.org/jira/browse/MESOS-6010
> Project: Mesos
> Issue Type: Bug
> Components: containerization, docker
>Affects Versions: 1.0.0, 1.0.1
>Reporter: Sunzhe
>Assignee: Jan Schlicht
>Priority: Critical
> Labels: Docker, mesos-containerizer
>
> The {{mesos-agent}} flags:
> {code}
> GLOG_v=1 ./bin/mesos-agent.sh \
> --master=zk://${MESOS_MASTER_IP}:2181/mesos \
> --ip=10.100.3.3 \
> --work_dir=${MESOS_WORK_DIR} \
>
> --isolation=cgroups/devices,gpu/nvidia,disk/du,docker/runtime,filesystem/linux
> \
> --enforce_container_disk_quota \
> --containerizers=mesos \
> --image_providers=docker \
> --executor_environment_variables="{}"
> {code}
> And the {{mesos-execute}} flags:
> {code}
> ./src/mesos-execute \
>--master=${MESOS_MASTER_IP}:5050 \
>--name=${INSTANCE_NAME} \
>--docker_image=${DOCKER_IMAGE} \
>--framework_capabilities=GPU_RESOURCES \
>--shell=false
> {code}
> But when {{./src/mesos-execute}}, the errors like below:
> {code}
> I0809 16:11:46.207875 25583 scheduler.cpp:172] Version: 1.0.0
> I0809 16:11:46.212442 25582 scheduler.cpp:461] New master detected at
> master@10.103.0.125:5050
> Subscribed with ID '168ab900-ee7e-4829-a59a-d16de956637e-0009'
> Submitted task 'test' to agent '168ab900-ee7e-4829-a59a-d16de956637e-S1'
> Received status update TASK_FAILED for task 'test'
> message: 'Failed to launch container: Failed to decode HTTP responses: No
> response decoded
> HTTP/1.1 200 Connection established
> HTTP/1.1 401 Unauthorized
> Content-Type: application/json; charset=utf-8
> Docker-Distribution-Api-Version: registry/2.0
> Www-Authenticate: Bearer
> realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/redis:pull";
> Date: Tue, 09 Aug 2016 08:10:32 GMT
> Content-Length: 145
> Strict-Tra
[jira] [Comment Edited] (MESOS-6010) Docker registry puller shows decode error "No response decoded".
[
https://issues.apache.org/jira/browse/MESOS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586271#comment-15586271
]
Gilbert Song edited comment on MESOS-6010 at 10/18/16 6:36 PM:
---
The root cause is that our 3rd party http_parser cannot handle the case that a
http response does not contain any header and body. It would return `No
response decoded`.
Currently, we are using `curl` to pull docker images. When we firstly send a
request to the registry server for manifest, we expect the http response like
the following:
{noformat}
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer
realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/redis:pull";
Date: Tue, 09 Aug 2016 08:10:32 GMT
Content-Length: 145
Strict-Transport-Security: max-age=31536000
{"errors":[{"code":"UNAUTHORIZED","message":"authentication
required","detail":[{"Type":"repository","Name":"library/redis","Action":"pull"}]}]}
{noformat}
However, when the agent node is behind a proxy. The http response returned from
curl may contain the proxy connect information as an extra http response (may
or may not container headers/body). Then, the response may be as following:
{noformat}
HTTP/1.1 200 Connection established
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer
realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/redis:pull";
Date: Tue, 09 Aug 2016 08:10:32 GMT
Content-Length: 145
Strict-Transport-Security: max-age=31536000
{noformat}
It is problematic for the 3rd party http_parser (currently we are using
http-parser-2.6.2) to parse a response without herders/body. We should either:
1. Fix the 3rd party http_parser library, or upgrade to a version contains the
fix, or find another substitute library.
2. Introduce a workaround to skip the response from the proxy (e.g., config the
curl using some flags to ignore the proxy response).
I would personally prefer (1), since we may have similar issues coming up in
the future.
was (Author: gilbert):
The root cause is that our 3rd party http_parser cannot handle the case that a
http response does not contain any header and body. It would return `No
response decoded`.
Currently, we are using `curl` to pull docker images. When we firstly send a
request to the registry server for manifest, we expect the http response like
the following:
{noformat}
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer
realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/redis:pull";
Date: Tue, 09 Aug 2016 08:10:32 GMT
Content-Length: 145
Strict-Transport-Security: max-age=31536000
{"errors":[{"code":"UNAUTHORIZED","message":"authentication
required","detail":[{"Type":"repository","Name":"library/redis","Action":"pull"}]}]}
{noformat}
However, when the agent node is behind a proxy. The http response returned from
curl may contain the proxy connect information as an extra http response (may
or may not container headers/body). Then, the response may be as following:
{noformat}
{noformat}
It is problematic for the 3rd party http_parser (currently we are using
http-parser-2.6.2) to parse a response without herders/body. We should either:
1. Fix the 3rd party http_parser library, or upgrade to a version contains the
fix, or find another substitute library.
2. Introduce a workaround to skip the response from the proxy (e.g., config the
curl using some flags to ignore the proxy response).
I would personally prefer (1), since we may have similar issues coming up in
the future.
> Docker registry puller shows decode error "No response decoded".
>
>
> Key: MESOS-6010
> URL: https://issues.apache.org/jira/browse/MESOS-6010
> Project: Mesos
> Issue Type: Bug
> Components: containerization, docker
>Affects Versions: 1.0.0, 1.0.1
>Reporter: Sunzhe
> Labels: Docker, mesos-containerizer
>
> The {{mesos-agent}} flags:
> {code}
> GLOG_v=1 ./bin/mesos-agent.sh \
> --master=zk://${MESOS_MASTER_IP}:2181/mesos \
> --ip=10.100.3.3 \
> --work_dir=${MESOS_WORK_DIR} \
>
> --isolation=cgroups/devices,gpu/nvidia,disk/du,docker/runtime,filesystem/linux
> \
> --enforce_container_disk_quota \
> --containerizers=mesos \
> --image_providers=docker \
> --executor_environment_variables="{}"
> {code}
> And the {{mesos-execute}} flags:
> {code}
> ./src/meso
[jira] [Comment Edited] (MESOS-6010) Docker registry puller shows decode error "No response decoded".
[
https://issues.apache.org/jira/browse/MESOS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586271#comment-15586271
]
Gilbert Song edited comment on MESOS-6010 at 10/18/16 6:35 PM:
---
The root cause is that our 3rd party http_parser cannot handle the case that a
http response does not contain any header and body. It would return `No
response decoded`.
Currently, we are using `curl` to pull docker images. When we firstly send a
request to the registry server for manifest, we expect the http response like
the following:
{noformat}
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer
realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/redis:pull";
Date: Tue, 09 Aug 2016 08:10:32 GMT
Content-Length: 145
Strict-Transport-Security: max-age=31536000
{"errors":[{"code":"UNAUTHORIZED","message":"authentication
required","detail":[{"Type":"repository","Name":"library/redis","Action":"pull"}]}]}
{noformat}
However, when the agent node is behind a proxy. The http response returned from
curl may contain the proxy connect information as an extra http response (may
or may not container headers/body). Then, the response may be as following:
{noformat}
{noformat}
It is problematic for the 3rd party http_parser (currently we are using
http-parser-2.6.2) to parse a response without herders/body. We should either:
1. Fix the 3rd party http_parser library, or upgrade to a version contains the
fix, or find another substitute library.
2. Introduce a workaround to skip the response from the proxy (e.g., config the
curl using some flags to ignore the proxy response).
I would personally prefer (1), since we may have similar issues coming up in
the future.
was (Author: gilbert):
The root cause is that our 3rd party http_parser cannot handle the case that a
http response does not contain any header and body. It would return `No
response decoded`.
Currently, we are using `curl` to pull docker images. When we firstly send a
request to the registry server for manifest, we expect the http response like
the following:
{noformat}
{noformat}
However, when the agent node is behind a proxy. The http response returned from
curl may contain the proxy connect information as an extra http response (may
or may not container headers/body). Then, the response may be as following:
{noformat}
{noformat}
It is problematic for the 3rd party http_parser (currently we are using
http-parser-2.6.2) to parse a response without herders/body. We should either:
1. Fix the 3rd party http_parser library, or upgrade to a version contains the
fix, or find another substitute library.
2. Introduce a workaround to skip the response from the proxy (e.g., config the
curl using some flags to ignore the proxy response).
I would personally prefer (1), since we may have similar issues coming up in
the future.
> Docker registry puller shows decode error "No response decoded".
>
>
> Key: MESOS-6010
> URL: https://issues.apache.org/jira/browse/MESOS-6010
> Project: Mesos
> Issue Type: Bug
> Components: containerization, docker
>Affects Versions: 1.0.0, 1.0.1
>Reporter: Sunzhe
> Labels: Docker, mesos-containerizer
>
> The {{mesos-agent}} flags:
> {code}
> GLOG_v=1 ./bin/mesos-agent.sh \
> --master=zk://${MESOS_MASTER_IP}:2181/mesos \
> --ip=10.100.3.3 \
> --work_dir=${MESOS_WORK_DIR} \
>
> --isolation=cgroups/devices,gpu/nvidia,disk/du,docker/runtime,filesystem/linux
> \
> --enforce_container_disk_quota \
> --containerizers=mesos \
> --image_providers=docker \
> --executor_environment_variables="{}"
> {code}
> And the {{mesos-execute}} flags:
> {code}
> ./src/mesos-execute \
>--master=${MESOS_MASTER_IP}:5050 \
>--name=${INSTANCE_NAME} \
>--docker_image=${DOCKER_IMAGE} \
>--framework_capabilities=GPU_RESOURCES \
>--shell=false
> {code}
> But when {{./src/mesos-execute}}, the errors like below:
> {code}
> I0809 16:11:46.207875 25583 scheduler.cpp:172] Version: 1.0.0
> I0809 16:11:46.212442 25582 scheduler.cpp:461] New master detected at
> master@10.103.0.125:5050
> Subscribed with ID '168ab900-ee7e-4829-a59a-d16de956637e-0009'
> Submitted task 'test' to agent '168ab900-ee7e-4829-a59a-d16de956637e-S1'
> Received status update TASK_FAILED for task 'test'
> message: 'Failed to launch container: Failed to decode HTTP responses: No
> response decoded
> HTTP/1.1 200 Connection established
> HTTP/1.1 401 Unauthorized
> Content-Type: application/json; charset=utf-8
> Docker
[jira] [Comment Edited] (MESOS-6010) Docker registry puller shows decode error "No response decoded".
[
https://issues.apache.org/jira/browse/MESOS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414576#comment-15414576
]
Sunzhe edited comment on MESOS-6010 at 8/10/16 1:52 AM:
The image is {{library/redis}}, and I didn't use the agent flag
{{-\-docker_registry}}, that is to say, I used the default docker registry. If
using the default Docker registry I was consistently getting this error.
But if I used the agent flag {{-\-docker_registry}} is a local
path(i.e:{{/tmp/docker/images}}) in which Docker image archives(result of
{{docker save}}) are stored, the {{mesos-execute}} works well.
was (Author: sunzhe):
The image is {{redis}}, and I didn't use the agent flag {{-\-docker_registry}},
that is to say, I used the default docker registry. If using the default Docker
registry I was consistently getting this error.
But if I used the agent flag {{-\-docker_registry}} is a local
path(i.e:{{/tmp/docker/images}}) in which Docker image archives(result of
{{docker save}}) are stored, the {{mesos-execute}} works well.
> Docker registry puller shows decode error "No response decoded".
>
>
> Key: MESOS-6010
> URL: https://issues.apache.org/jira/browse/MESOS-6010
> Project: Mesos
> Issue Type: Bug
> Components: containerization, docker
>Affects Versions: 1.0.0
>Reporter: Sunzhe
> Labels: Docker, mesos-containerizer
>
> The {{mesos-agent}} flags:
> {code}
> GLOG_v=1 ./bin/mesos-agent.sh \
> --master=zk://${MESOS_MASTER_IP}:2181/mesos \
> --ip=10.100.3.3 \
> --work_dir=${MESOS_WORK_DIR} \
>
> --isolation=cgroups/devices,gpu/nvidia,disk/du,docker/runtime,filesystem/linux
> \
> --enforce_container_disk_quota \
> --containerizers=mesos \
> --image_providers=docker \
> --executor_environment_variables="{}"
> {code}
> And the {{mesos-execute}} flags:
> {code}
> ./src/mesos-execute \
>--master=${MESOS_MASTER_IP}:5050 \
>--name=${INSTANCE_NAME} \
>--docker_image=${DOCKER_IMAGE} \
>--framework_capabilities=GPU_RESOURCES \
>--shell=false
> {code}
> But when {{./src/mesos-execute}}, the errors like below:
> {code}
> I0809 16:11:46.207875 25583 scheduler.cpp:172] Version: 1.0.0
> I0809 16:11:46.212442 25582 scheduler.cpp:461] New master detected at
> master@10.103.0.125:5050
> Subscribed with ID '168ab900-ee7e-4829-a59a-d16de956637e-0009'
> Submitted task 'test' to agent '168ab900-ee7e-4829-a59a-d16de956637e-S1'
> Received status update TASK_FAILED for task 'test'
> message: 'Failed to launch container: Failed to decode HTTP responses: No
> response decoded
> HTTP/1.1 200 Connection established
> HTTP/1.1 401 Unauthorized
> Content-Type: application/json; charset=utf-8
> Docker-Distribution-Api-Version: registry/2.0
> Www-Authenticate: Bearer
> realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/redis:pull";
> Date: Tue, 09 Aug 2016 08:10:32 GMT
> Content-Length: 145
> Strict-Transport-Security: max-age=31536000
> {"errors":[{"code":"UNAUTHORIZED","message":"authentication
> required","detail":[{"Type":"repository","Name":"library/redis","Action":"pull"}]}]}
> ; Container destroyed while provisioning images'
> source: SOURCE_AGENT
> reason: REASON_CONTAINER_LAUNCH_FAILED
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (MESOS-6010) Docker registry puller shows decode error "No response decoded".
[
https://issues.apache.org/jira/browse/MESOS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414576#comment-15414576
]
Sunzhe edited comment on MESOS-6010 at 8/10/16 1:47 AM:
The image is {{redis}}, and I didn't use the agent flag {{-\-docker_registry}},
that is to say, I used the default docker registry. If using the default Docker
registry I was consistently getting this error.
But if I used the agent flag {{-\-docker_registry}} is a local
path(i.e:{{/tmp/docker/images}}) in which Docker image archives(result of
{{docker save}}) are stored, the {{mesos-execute}} works well.
was (Author: sunzhe):
The image is {{redis}}, and I didn't use the flag {{-\-docker_registry}}, that
is to say, I used the default docker registry. If using the default Docker
registry I was consistently getting this error.
But if I used the flag {{-\-docker_registry}} is a local
path(i.e:{{/tmp/docker/images}}) in which Docker image archives(result of
{{docker save}}) are stored, the {{mesos-execute}} works well.
> Docker registry puller shows decode error "No response decoded".
>
>
> Key: MESOS-6010
> URL: https://issues.apache.org/jira/browse/MESOS-6010
> Project: Mesos
> Issue Type: Bug
> Components: containerization, docker
>Affects Versions: 1.0.0
>Reporter: Sunzhe
> Labels: Docker, mesos-containerizer
>
> The {{mesos-agent}} flags:
> {code}
> GLOG_v=1 ./bin/mesos-agent.sh \
> --master=zk://${MESOS_MASTER_IP}:2181/mesos \
> --ip=10.100.3.3 \
> --work_dir=${MESOS_WORK_DIR} \
>
> --isolation=cgroups/devices,gpu/nvidia,disk/du,docker/runtime,filesystem/linux
> \
> --enforce_container_disk_quota \
> --containerizers=mesos \
> --image_providers=docker \
> --executor_environment_variables="{}"
> {code}
> And the {{mesos-execute}} flags:
> {code}
> ./src/mesos-execute \
>--master=${MESOS_MASTER_IP}:5050 \
>--name=${INSTANCE_NAME} \
>--docker_image=${DOCKER_IMAGE} \
>--framework_capabilities=GPU_RESOURCES \
>--shell=false
> {code}
> But when {{./src/mesos-execute}}, the errors like below:
> {code}
> I0809 16:11:46.207875 25583 scheduler.cpp:172] Version: 1.0.0
> I0809 16:11:46.212442 25582 scheduler.cpp:461] New master detected at
> master@10.103.0.125:5050
> Subscribed with ID '168ab900-ee7e-4829-a59a-d16de956637e-0009'
> Submitted task 'test' to agent '168ab900-ee7e-4829-a59a-d16de956637e-S1'
> Received status update TASK_FAILED for task 'test'
> message: 'Failed to launch container: Failed to decode HTTP responses: No
> response decoded
> HTTP/1.1 200 Connection established
> HTTP/1.1 401 Unauthorized
> Content-Type: application/json; charset=utf-8
> Docker-Distribution-Api-Version: registry/2.0
> Www-Authenticate: Bearer
> realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/redis:pull";
> Date: Tue, 09 Aug 2016 08:10:32 GMT
> Content-Length: 145
> Strict-Transport-Security: max-age=31536000
> {"errors":[{"code":"UNAUTHORIZED","message":"authentication
> required","detail":[{"Type":"repository","Name":"library/redis","Action":"pull"}]}]}
> ; Container destroyed while provisioning images'
> source: SOURCE_AGENT
> reason: REASON_CONTAINER_LAUNCH_FAILED
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (MESOS-6010) Docker registry puller shows decode error "No response decoded".
[
https://issues.apache.org/jira/browse/MESOS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414576#comment-15414576
]
Sunzhe edited comment on MESOS-6010 at 8/10/16 1:41 AM:
The image is {{redis}}, and I didn't use the flag {{-\-docker_registry}}, that
is to say, I used the default docker registry. If using the default Docker
registry I was consistently getting this error.
But if I used the flag {{-\-docker_registry}} is a local
path(i.e:{{/tmp/docker/images}}) in which Docker image archives(result of
{{docker save}}) are stored, the {{mesos-execute}} works well.
was (Author: sunzhe):
The image is {{redis}}, and I didn't use the flag {{-\-docker_registry}}, that
is to say, I used the default docker registry. If using the default Docker
registry I was consistently getting this error.
But if I used the flag {{-\-docker_registry}} is a local
path(i.e:/tmp/docker/images) in which Docker image archives(result of {{docker
save}}) are stored, the {{mesos-execute}} works well.
> Docker registry puller shows decode error "No response decoded".
>
>
> Key: MESOS-6010
> URL: https://issues.apache.org/jira/browse/MESOS-6010
> Project: Mesos
> Issue Type: Bug
> Components: containerization, docker
>Affects Versions: 1.0.0
>Reporter: Sunzhe
> Labels: Docker, mesos-containerizer
>
> The {{mesos-agent}} flags:
> {code}
> GLOG_v=1 ./bin/mesos-agent.sh \
> --master=zk://${MESOS_MASTER_IP}:2181/mesos \
> --ip=10.100.3.3 \
> --work_dir=${MESOS_WORK_DIR} \
>
> --isolation=cgroups/devices,gpu/nvidia,disk/du,docker/runtime,filesystem/linux
> \
> --enforce_container_disk_quota \
> --containerizers=mesos \
> --image_providers=docker \
> --executor_environment_variables="{}"
> {code}
> And the {{mesos-execute}} flags:
> {code}
> ./src/mesos-execute \
>--master=${MESOS_MASTER_IP}:5050 \
>--name=${INSTANCE_NAME} \
>--docker_image=${DOCKER_IMAGE} \
>--framework_capabilities=GPU_RESOURCES \
>--shell=false
> {code}
> But when {{./src/mesos-execute}}, the errors like below:
> {code}
> I0809 16:11:46.207875 25583 scheduler.cpp:172] Version: 1.0.0
> I0809 16:11:46.212442 25582 scheduler.cpp:461] New master detected at
> master@10.103.0.125:5050
> Subscribed with ID '168ab900-ee7e-4829-a59a-d16de956637e-0009'
> Submitted task 'test' to agent '168ab900-ee7e-4829-a59a-d16de956637e-S1'
> Received status update TASK_FAILED for task 'test'
> message: 'Failed to launch container: Failed to decode HTTP responses: No
> response decoded
> HTTP/1.1 200 Connection established
> HTTP/1.1 401 Unauthorized
> Content-Type: application/json; charset=utf-8
> Docker-Distribution-Api-Version: registry/2.0
> Www-Authenticate: Bearer
> realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/redis:pull";
> Date: Tue, 09 Aug 2016 08:10:32 GMT
> Content-Length: 145
> Strict-Transport-Security: max-age=31536000
> {"errors":[{"code":"UNAUTHORIZED","message":"authentication
> required","detail":[{"Type":"repository","Name":"library/redis","Action":"pull"}]}]}
> ; Container destroyed while provisioning images'
> source: SOURCE_AGENT
> reason: REASON_CONTAINER_LAUNCH_FAILED
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
