[
https://issues.apache.org/jira/browse/YARN-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated YARN-2185:
---------------------------------
Attachment: YARN-2185.000.patch
Attaching my suggestion how to solve this. The code streams HDFS as standard
input to the tar and gzip commands. It handles Windows as well. As an addition
I create temporary files with permissions 700 instead of 755. I do not create
any additional temporary directories for extraction, one is enough. A
difference is that I use jar command for zips as well, so that it handles
Windows properly. Also I added an additional switch to be able to disable the
modification time check specifying -1 as the timestamp.
> Use pipes when localizing archives
> ----------------------------------
>
> Key: YARN-2185
> URL: https://issues.apache.org/jira/browse/YARN-2185
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.4.0
> Reporter: Jason Lowe
> Assignee: Miklos Szegedi
> Attachments: YARN-2185.000.patch
>
>
> Currently the nodemanager downloads an archive to a local file, unpacks it,
> and then removes it. It would be more efficient to stream the data as it's
> being unpacked to avoid both the extra disk space requirements and the
> additional disk activity from storing the archive.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
