[
https://issues.apache.org/jira/browse/MESOS-6587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gilbert Song updated MESOS-6587:
--------------------------------
Description:
When caching an archive that is retrieved from a signed URL from somewhere like
Amazon S3 or Azure Blob, the archive is not decompressed properly even when a
valid compression suffix is set on the output_file parameter.
An example log is show below:
{noformat}
I1114 14:49:49.689990 39178 logging.cpp:194] INFO level logging started!
I1114 14:49:49.690237 39178 fetcher.cpp:498] Fetcher Info:
{"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/68721b22-f102-443a-887c-b1df78f40bf5-S8\/root","items":[{"action":"RETRIEVE_FROM_CACHE","cache_filename":"c4-docker.tar_5.255&sr=b","uri":{"cache":true,"executable":false,"extract":true,"output_file":"docker.tar.gz","value":"https:\/\/reportresources.blob.core.windows.net\/mesos\/docker.tar.gz?sig=thesignaturegoeshere"}}],"sandbox_directory":"\/var\/lib\/mesos\/slave\/slaves\/68721b22-f102-443a-887c-b1df78f40bf5-S8\/frameworks\/68721b22-f102-443a-887c-b1df78f40bf5-0000\/executors\/test.97c76288-aa79-11e6-9316-70b3d5800002\/runs\/a21ecf01-e80a-4d2b-b094-34d442081818","user":"root"}
I1114 14:49:49.692350 39178 fetcher.cpp:409] Fetching URI
'https://reportresources.blob.core.windows.net/mesos/docker.tar.gz?sig=thesignaturegoeshere'
I1114 14:49:49.692369 39178 fetcher.cpp:306] Fetching from cache
W1114 14:49:49.692384 39178 fetcher.cpp:350] Copying instead of extracting
resource from URI with 'extract' flag, because it does not seem to be an
archive:
https://reportresources.blob.core.windows.net/mesos/docker.tar.gz?sig=thesignaturegoeshere
I1114 14:49:49.692464 39178 fetcher.cpp:167] Copying resource with command:cp
'/tmp/mesos/fetch/slaves/68721b22-f102-443a-887c-b1df78f40bf5-S8/root/c4-docker.tar_5.255&sr=b'
'/var/lib/mesos/slave/slaves/68721b22-f102-443a-887c-b1df78f40bf5-S8/frameworks/68721b22-f102-443a-887c-b1df78f40bf5-0000/executors/test.97c76288-aa79-11e6-9316-70b3d5800002/runs/a21ecf01-e80a-4d2b-b094-34d442081818/docker.tar.gz'
I1114 14:49:49.694368 39178 fetcher.cpp:547] Fetched
'https://reportresources.blob.core.windows.net/mesos/docker.tar.gz?sig=thesignaturegoeshere'
to
'/var/lib/mesos/slave/slaves/68721b22-f102-443a-887c-b1df78f40bf5-S8/frameworks/68721b22-f102-443a-887c-b1df78f40bf5-0000/executors/test.97c76288-aa79-11e6-9316-70b3d5800002/runs/a21ecf01-e80a-4d2b-b094-34d442081818/docker.tar.gz'
{noformat}
Even though the output_file is set to docker.tar.gz, the archive is copied
instead of extracted because of the signature suffix from the source URL.
was:
When caching an archive that is retrieved from a signed URL from somewhere like
Amazon S3 or Azure Blob, the archive is not decompressed properly even when a
valid compression suffix is set on the output_file parameter.
An example log is show below:
I1114 14:49:49.689990 39178 logging.cpp:194] INFO level logging started!
I1114 14:49:49.690237 39178 fetcher.cpp:498] Fetcher Info:
{"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/68721b22-f102-443a-887c-b1df78f40bf5-S8\/root","items":[{"action":"RETRIEVE_FROM_CACHE","cache_filename":"c4-docker.tar_5.255&sr=b","uri":{"cache":true,"executable":false,"extract":true,"output_file":"docker.tar.gz","value":"https:\/\/reportresources.blob.core.windows.net\/mesos\/docker.tar.gz?sig=thesignaturegoeshere"}}],"sandbox_directory":"\/var\/lib\/mesos\/slave\/slaves\/68721b22-f102-443a-887c-b1df78f40bf5-S8\/frameworks\/68721b22-f102-443a-887c-b1df78f40bf5-0000\/executors\/test.97c76288-aa79-11e6-9316-70b3d5800002\/runs\/a21ecf01-e80a-4d2b-b094-34d442081818","user":"root"}
I1114 14:49:49.692350 39178 fetcher.cpp:409] Fetching URI
'https://reportresources.blob.core.windows.net/mesos/docker.tar.gz?sig=thesignaturegoeshere'
I1114 14:49:49.692369 39178 fetcher.cpp:306] Fetching from cache
W1114 14:49:49.692384 39178 fetcher.cpp:350] Copying instead of extracting
resource from URI with 'extract' flag, because it does not seem to be an
archive:
https://reportresources.blob.core.windows.net/mesos/docker.tar.gz?sig=thesignaturegoeshere
I1114 14:49:49.692464 39178 fetcher.cpp:167] Copying resource with command:cp
'/tmp/mesos/fetch/slaves/68721b22-f102-443a-887c-b1df78f40bf5-S8/root/c4-docker.tar_5.255&sr=b'
'/var/lib/mesos/slave/slaves/68721b22-f102-443a-887c-b1df78f40bf5-S8/frameworks/68721b22-f102-443a-887c-b1df78f40bf5-0000/executors/test.97c76288-aa79-11e6-9316-70b3d5800002/runs/a21ecf01-e80a-4d2b-b094-34d442081818/docker.tar.gz'
I1114 14:49:49.694368 39178 fetcher.cpp:547] Fetched
'https://reportresources.blob.core.windows.net/mesos/docker.tar.gz?sig=thesignaturegoeshere'
to
'/var/lib/mesos/slave/slaves/68721b22-f102-443a-887c-b1df78f40bf5-S8/frameworks/68721b22-f102-443a-887c-b1df78f40bf5-0000/executors/test.97c76288-aa79-11e6-9316-70b3d5800002/runs/a21ecf01-e80a-4d2b-b094-34d442081818/docker.tar.gz'
Even though the output_file is set to docker.tar.gz, the archive is copied
instead of extracted because of the signature suffix from the source URL.
> Unable to cache certain archives with custom output_file
> --------------------------------------------------------
>
> Key: MESOS-6587
> URL: https://issues.apache.org/jira/browse/MESOS-6587
> Project: Mesos
> Issue Type: Bug
> Components: fetcher
> Reporter: Stephen Hankinson
> Assignee: Stephen Hankinson
> Priority: Minor
>
> When caching an archive that is retrieved from a signed URL from somewhere
> like Amazon S3 or Azure Blob, the archive is not decompressed properly even
> when a valid compression suffix is set on the output_file parameter.
> An example log is show below:
> {noformat}
> I1114 14:49:49.689990 39178 logging.cpp:194] INFO level logging started!
> I1114 14:49:49.690237 39178 fetcher.cpp:498] Fetcher Info:
> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/68721b22-f102-443a-887c-b1df78f40bf5-S8\/root","items":[{"action":"RETRIEVE_FROM_CACHE","cache_filename":"c4-docker.tar_5.255&sr=b","uri":{"cache":true,"executable":false,"extract":true,"output_file":"docker.tar.gz","value":"https:\/\/reportresources.blob.core.windows.net\/mesos\/docker.tar.gz?sig=thesignaturegoeshere"}}],"sandbox_directory":"\/var\/lib\/mesos\/slave\/slaves\/68721b22-f102-443a-887c-b1df78f40bf5-S8\/frameworks\/68721b22-f102-443a-887c-b1df78f40bf5-0000\/executors\/test.97c76288-aa79-11e6-9316-70b3d5800002\/runs\/a21ecf01-e80a-4d2b-b094-34d442081818","user":"root"}
> I1114 14:49:49.692350 39178 fetcher.cpp:409] Fetching URI
> 'https://reportresources.blob.core.windows.net/mesos/docker.tar.gz?sig=thesignaturegoeshere'
> I1114 14:49:49.692369 39178 fetcher.cpp:306] Fetching from cache
> W1114 14:49:49.692384 39178 fetcher.cpp:350] Copying instead of extracting
> resource from URI with 'extract' flag, because it does not seem to be an
> archive:
> https://reportresources.blob.core.windows.net/mesos/docker.tar.gz?sig=thesignaturegoeshere
> I1114 14:49:49.692464 39178 fetcher.cpp:167] Copying resource with command:cp
> '/tmp/mesos/fetch/slaves/68721b22-f102-443a-887c-b1df78f40bf5-S8/root/c4-docker.tar_5.255&sr=b'
>
> '/var/lib/mesos/slave/slaves/68721b22-f102-443a-887c-b1df78f40bf5-S8/frameworks/68721b22-f102-443a-887c-b1df78f40bf5-0000/executors/test.97c76288-aa79-11e6-9316-70b3d5800002/runs/a21ecf01-e80a-4d2b-b094-34d442081818/docker.tar.gz'
> I1114 14:49:49.694368 39178 fetcher.cpp:547] Fetched
> 'https://reportresources.blob.core.windows.net/mesos/docker.tar.gz?sig=thesignaturegoeshere'
> to
> '/var/lib/mesos/slave/slaves/68721b22-f102-443a-887c-b1df78f40bf5-S8/frameworks/68721b22-f102-443a-887c-b1df78f40bf5-0000/executors/test.97c76288-aa79-11e6-9316-70b3d5800002/runs/a21ecf01-e80a-4d2b-b094-34d442081818/docker.tar.gz'
> {noformat}
> Even though the output_file is set to docker.tar.gz, the archive is copied
> instead of extracted because of the signature suffix from the source URL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
