[
https://issues.apache.org/jira/browse/MESOS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105988#comment-15105988
]
Erik Weathers commented on MESOS-3367:
--------------------------------------
Thanks for the attention on this issue [[email protected]] & [~bernd-mesos]!
It's not clear to me how the patches that [[email protected]] posted would
help solve the issue.
Also, the lack of this feature prevents using mesos's fetcher to grab tarballs
from WebHDFS, since when you download a file over WebHDFS the resultant URL
includes a suffix like
{{?op=OPEN&namenoderpcaddress=hadoop-namenode&offset=0}}, and with an HTTP
Content-Type of {{application/octet-stream}}. Luckily the Linux {{file}}
command reveals the true nature of the files:
{code}
% wget
http://hadoop-namenode:50070/webhdfs/v1/user/erikdw/storm-mesos-0.9.6.tgz?op=OPEN
% mv
'storm-mesos-0.9.6.tgz?op=OPEN&namenoderpcaddress=hadoop-namenode&offset=0'
foo.tar.gz
% file foo.tar.gz
foo.tar.gz: gzip compressed data, from Unix, last modified: Mon Jan 18 22:30:57
2016
% unzip foo.tar.gz
% file foo.tar
foo.tar: POSIX tar archive (GNU)
{code}
NOTE: Unfortunately, {{gzip}} is kind of simplistic and ["cowardly refuses" (a
la
tar)|https://github.com/kikitux/tar/blob/5e2a1d5b3801d016f51b3f4c476d275a6adff5d7/src/tar.c#L2891]
to uncompress a file with an unexpected suffix, hence my renaming above.
Luckily {{tar}} is more robust and can directly extract such a file without
renaming. But... unfortunately, {{tar}} fails if such a "gzip compressed data"
file is not *also* a tar file. Sigh.
> Mesos fetcher does not extract archives for URI with parameters
> ---------------------------------------------------------------
>
> Key: MESOS-3367
> URL: https://issues.apache.org/jira/browse/MESOS-3367
> Project: Mesos
> Issue Type: Improvement
> Components: fetcher
> Affects Versions: 0.22.1, 0.23.0
> Environment: DCOS 1.1
> Reporter: Renat Zubairov
> Assignee: haosdent
> Priority: Minor
> Labels: mesosphere
>
> I'm deploying using marathon applications with sources served from S3. I'm
> using a signed URL to give only temporary access to the S3 resources, so URL
> of the resource have some query parameters.
> So URI is 'https://foo.com/file.tgz?hasi' and fetcher stores it in the file
> with the name 'file.tgz?hasi', then it thinks that extension 'hasi' is not
> tgz hence extraction is skipped, despite the fact that MIME Type of the HTTP
> resource is 'application/x-tar'.
> Workaround - add additional parameter like '&workaround=.tgz'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
