[
https://issues.apache.org/jira/browse/HADOOP-10948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596780#comment-16596780
]
Shyamsundar Ramanathan commented on HADOOP-10948:
-------------------------------------------------
* We applied the patch and have noticed a significant degradation in
performance, specifically in listing files (may affect other areas too). * We
found that in the patch, an additional head call is made to each subpath when
listing a dir (to determine if the manifest header exists etc).
* This significantly affects performance especially if the number of files in
a particular "directory" are large. (few seconds without the patch vs minutes
with the patch for nFiles > 500~1000) (due to increased number of rest calls)
Specifically these lines cause the issue:
{code}
+ if (!name.endsWith("/")) {
+ final Path filePath = getCorrectSwiftPath(new Path(name));
+ files.add(getObjectMetadata(filePath, newest));
+ } else {
+ final Path dirPath = getCorrectSwiftPath(toDirPath(new Path(name)));
+ files.add(getObjectMetadata(dirPath, newest));
}
{code}
> SwiftNativeFileSystem's directory is incompatible with Swift and Horizon
> ------------------------------------------------------------------------
>
> Key: HADOOP-10948
> URL: https://issues.apache.org/jira/browse/HADOOP-10948
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/swift
> Affects Versions: 3.0.0-alpha1
> Reporter: Kazuki OIKAWA
> Assignee: Kazuki OIKAWA
> Priority: Major
> Labels: BB2015-05-TBR
> Attachments: HADOOP-10948-2.patch, HADOOP-10948.patch
>
>
> SwiftNativeFileSystem's directory representation is zero-byte file.
> But in Swift / Horizon, directory representation is a trailing-slash.
> This incompatibility has the following issues.
> * SwiftNativeFileSystem can't see pseudo-directory made by OpenStack Horizon
> * Swift/Horizon can't see pseudo-directory made by SwiftNativeFileSystem. But
> Swift/Horizon see a zero-byte file instead of that pseudo-directory.
> * SwiftNativeFileSystem can't see a file if there is no intermediate
> pseudo-directory object.
> * SwiftNativeFileSystem makes two objects when making a single directory
> (e.g. "hadoop fs -mkdir swift://test.test/dir/" => "dir" and "dir/" created)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
