[
https://issues.apache.org/jira/browse/MESOS-7621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Allen updated MESOS-7621:
---------------------------------
Description:
{code}
$ curl -L -v -O -s http://HOSTNAME_REDACTED/PATH_REDACTED.tar.gz
* Trying 172.17.4.10...
* Connected to HOSTNAME_REDACTED (172.17.4.10) port 80 (#0)
> GET /PATH_REDACTED.tar.gz HTTP/1.1
> Host: HOSTNAME_REDACTED
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 302 FOUND
< Server: nginx/1.4.6 (Ubuntu)
< Date: Mon, 05 Jun 2017 17:58:04 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 1947
< Connection: keep-alive
< Location:
https://BUCKET_REDACTED.s3.amazonaws.com:443/PATH_REDACTED?Signature=REDACTED%3D&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D
<
* Ignoring the response-body
{ [309 bytes data]
* Connection #0 to host HOSTNAME_REDACTED left intact
* Issue another request to this URL:
'https://BUCKET_REDACTED.s3.amazonaws.com:443/PATH_REDACTED.tar.gz?Signature=SIGNATURE_REDACTED%3D&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D'
* Trying 54.231.40.75...
* Connected to BUCKET_REDACTED.s3.amazonaws.com (54.231.40.75) port 443 (#1)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.s3.amazonaws.com
* Server certificate: DigiCert Baltimore CA-2 G2
* Server certificate: Baltimore CyberTrust Root
> GET
> /PATH_REDACTED.tar.gz?Signature=REDACTED&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D
> HTTP/1.1
> Host: BUCKET_REDACTED.s3.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< x-amz-id-2: ID_REDACTED=
< x-amz-request-id: REQUEST_ID_REDACTED
< Date: Mon, 05 Jun 2017 17:58:07 GMT
< Last-Modified: Thu, 01 Jun 2017 03:04:49 GMT
< ETag: "ETAG_REDACTED"
< Accept-Ranges: bytes
< Content-Type: application/x-tar
< Content-Length: 208245664
< Server: AmazonS3
<
{ [16360 bytes data]
{code}
We have a micro-service which signs temporary urls for services which can't
speak natively with S3. The above is an example download using {{curl}}. But
when using the mesos fetcher the agent logs contain the following information:
{code}
fetcher.cpp:479] Reverting to fetching directly into the sandbox for
'http://HOST_REDACTED/PATH_REDACTED.tar.gz', due to failure to fetch through
the cache, with error: Could not determine size of cache file for
'USER_REDACTED@http://HOST_REDACTED/PATH_REDACTED.tar.gz' with error: No URL
content-length available
{code}
After some more investigation, it looks like the signing service had a bug
where it would not sign the redirect URL for HEAD vs GET (only GET) so the HEAD
request would fail with a 403. The source code erroneously reports this as
missing Content-Length rather than a 403 error on the HEAD request.
was:
{code}
$ curl -L -v -O -s http://HOSTNAME_REDACTED/PATH_REDACTED.tar.gz
* Trying 172.17.4.10...
* Connected to HOSTNAME_REDACTED (172.17.4.10) port 80 (#0)
> GET /PATH_REDACTED.tar.gz HTTP/1.1
> Host: HOSTNAME_REDACTED
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 302 FOUND
< Server: nginx/1.4.6 (Ubuntu)
< Date: Mon, 05 Jun 2017 17:58:04 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 1947
< Connection: keep-alive
< Location:
https://BUCKET_REDACTED.s3.amazonaws.com:443/PATH_REDACTED?Signature=REDACTED%3D&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D
<
* Ignoring the response-body
{ [309 bytes data]
* Connection #0 to host HOSTNAME_REDACTED left intact
* Issue another request to this URL:
'https://BUCKET_REDACTED.s3.amazonaws.com:443/PATH_REDACTED.tar.gz?Signature=SIGNATURE_REDACTED%3D&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D'
* Trying 54.231.40.75...
* Connected to BUCKET_REDACTED.s3.amazonaws.com (54.231.40.75) port 443 (#1)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.s3.amazonaws.com
* Server certificate: DigiCert Baltimore CA-2 G2
* Server certificate: Baltimore CyberTrust Root
> GET
> /PATH_REDACTED.tar.gz?Signature=REDACTED&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D
> HTTP/1.1
> Host: BUCKET_REDACTED.s3.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< x-amz-id-2: ID_REDACTED=
< x-amz-request-id: REQUEST_ID_REDACTED
< Date: Mon, 05 Jun 2017 17:58:07 GMT
< Last-Modified: Thu, 01 Jun 2017 03:04:49 GMT
< ETag: "ETAG_REDACTED"
< Accept-Ranges: bytes
< Content-Type: application/x-tar
< Content-Length: 208245664
< Server: AmazonS3
<
{ [16360 bytes data]
{code}
We have a micro-service which signs temporary urls for services which can't
speak natively with S3. The above is an example download using {{curl}}. But
when using the mesos fetcher the agent logs contain the following information:
{code}
fetcher.cpp:479] Reverting to fetching directly into the sandbox for
'http://HOST_REDACTED/PATH_REDACTED.tar.gz', due to failure to fetch through
the cache, with error: Could not determine size of cache file for
'USER_REDACTED@http://HOST_REDACTED/PATH_REDACTED.tar.gz' with error: No URL
content-length available
{code}
Any idea why this error would occur?
> Fetcher has unclear error message on HEAD fetch stage or cache sizing
> ---------------------------------------------------------------------
>
> Key: MESOS-7621
> URL: https://issues.apache.org/jira/browse/MESOS-7621
> Project: Mesos
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 1.2.0
> Reporter: Charles Allen
>
> {code}
> $ curl -L -v -O -s http://HOSTNAME_REDACTED/PATH_REDACTED.tar.gz
> * Trying 172.17.4.10...
> * Connected to HOSTNAME_REDACTED (172.17.4.10) port 80 (#0)
> > GET /PATH_REDACTED.tar.gz HTTP/1.1
> > Host: HOSTNAME_REDACTED
> > User-Agent: curl/7.43.0
> > Accept: */*
> >
> < HTTP/1.1 302 FOUND
> < Server: nginx/1.4.6 (Ubuntu)
> < Date: Mon, 05 Jun 2017 17:58:04 GMT
> < Content-Type: text/html; charset=utf-8
> < Content-Length: 1947
> < Connection: keep-alive
> < Location:
> https://BUCKET_REDACTED.s3.amazonaws.com:443/PATH_REDACTED?Signature=REDACTED%3D&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D
> <
> * Ignoring the response-body
> { [309 bytes data]
> * Connection #0 to host HOSTNAME_REDACTED left intact
> * Issue another request to this URL:
> 'https://BUCKET_REDACTED.s3.amazonaws.com:443/PATH_REDACTED.tar.gz?Signature=SIGNATURE_REDACTED%3D&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D'
> * Trying 54.231.40.75...
> * Connected to BUCKET_REDACTED.s3.amazonaws.com (54.231.40.75) port 443 (#1)
> * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
> * Server certificate: *.s3.amazonaws.com
> * Server certificate: DigiCert Baltimore CA-2 G2
> * Server certificate: Baltimore CyberTrust Root
> > GET
> > /PATH_REDACTED.tar.gz?Signature=REDACTED&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D
> > HTTP/1.1
> > Host: BUCKET_REDACTED.s3.amazonaws.com
> > User-Agent: curl/7.43.0
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < x-amz-id-2: ID_REDACTED=
> < x-amz-request-id: REQUEST_ID_REDACTED
> < Date: Mon, 05 Jun 2017 17:58:07 GMT
> < Last-Modified: Thu, 01 Jun 2017 03:04:49 GMT
> < ETag: "ETAG_REDACTED"
> < Accept-Ranges: bytes
> < Content-Type: application/x-tar
> < Content-Length: 208245664
> < Server: AmazonS3
> <
> { [16360 bytes data]
> {code}
> We have a micro-service which signs temporary urls for services which can't
> speak natively with S3. The above is an example download using {{curl}}. But
> when using the mesos fetcher the agent logs contain the following information:
> {code}
> fetcher.cpp:479] Reverting to fetching directly into the sandbox for
> 'http://HOST_REDACTED/PATH_REDACTED.tar.gz', due to failure to fetch through
> the cache, with error: Could not determine size of cache file for
> 'USER_REDACTED@http://HOST_REDACTED/PATH_REDACTED.tar.gz' with error: No URL
> content-length available
> {code}
> After some more investigation, it looks like the signing service had a bug
> where it would not sign the redirect URL for HEAD vs GET (only GET) so the
> HEAD request would fail with a 403. The source code erroneously reports this
> as missing Content-Length rather than a 403 error on the HEAD request.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
