[
https://issues.apache.org/jira/browse/HIVE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651394#comment-15651394
]
Steve Loughran commented on HIVE-15093:
---------------------------------------
-1 (non binding)
Doing parallel rename is a stop-gap solution which will be obsolete the moment
someone sits down to do it in s3a with an implementation that its more
efficient in its scheduling of copies calls, and, with tests and broader use,
more tested.
HADOOP-13600 proposes parallel renames. Nobody has written that yet, —but I
promise to review a patch people provide, with tests. Get that patch into
Hadoop and there's only one place to maintain this stuff, no need to
document/test another switch, maintain the option, have another codepath to
keep alive, etc.
The algorithm I proposed there would initially sorts the files by size, so the
larger renames are scheduled first. Given a thread pool smaller than the list
of files to rename, this should ensure that the scheduling is more optimal. the
listing. If you really, really, want to do this in a separate piece of code,
you should do the same.
Also, there are enough other s3a speedups that you should be testing against
Hadoop 2.8+, both to avoid optimising against a now-obsolete codepath, but also
to help find and report any problems in our code.
To summarise: go on, fix the code in Hadoop, simplify everyone's lives.
> S3-to-S3 Renames: Files should be moved individually rather than at a
> directory level
> -------------------------------------------------------------------------------------
>
> Key: HIVE-15093
> URL: https://issues.apache.org/jira/browse/HIVE-15093
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
> Affects Versions: 2.1.0
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Attachments: HIVE-15093.1.patch, HIVE-15093.2.patch,
> HIVE-15093.3.patch, HIVE-15093.4.patch, HIVE-15093.5.patch,
> HIVE-15093.6.patch, HIVE-15093.7.patch, HIVE-15093.8.patch, HIVE-15093.9.patch
>
>
> Hive's MoveTask uses the Hive.moveFile method to move data within a
> distributed filesystem as well as blobstore filesystems.
> If the move is done within the same filesystem:
> 1: If the source path is a subdirectory of the destination path, files will
> be moved one by one using a threapool of workers
> 2: If the source path is not a subdirectory of the destination path, a single
> rename operation is used to move the entire directory
> The second option may not work well on blobstores such as S3. Renames are not
> metadata operations and require copying all the data. Client connectors to
> blobstores may not efficiently rename directories. Worst case, the connector
> will copy each file one by one, sequentially rather than using a threadpool
> of workers to copy the data (e.g. HADOOP-13600).
> Hive already has code to rename files using a threadpool of workers, but this
> only occurs in case number 1.
> This JIRA aims to modify the code so that case 1 is triggered when copying
> within a blobstore. The focus is on copies within a blobstore because
> needToCopy will return true if the src and target filesystems are different,
> in which case a different code path is triggered.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
