[
https://issues.apache.org/jira/browse/HADOOP-11556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310682#comment-14310682
]
Steve Loughran commented on HADOOP-11556:
-----------------------------------------
the partitioned bit of swift is something that is best described as
"troublesome". It appears to violate some of the core semantics of a filesystem
(size of a file in a directory listing == size of file in stat() == amount of
data you can read), and things like rename get very confused.
Declaring swiftfs an object store with expensive renames via HADOOP-9565, along
with your fs -put improvements could somewhat improve this situation.
I believe that swift now supports >5GB files...adding support for that would
also help
> SwiftNativeFileSystem rename being interrupted during "fs -put"
> ---------------------------------------------------------------
>
> Key: HADOOP-11556
> URL: https://issues.apache.org/jira/browse/HADOOP-11556
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/swift
> Affects Versions: 2.6.0
> Reporter: Lei (Eddy) Xu
>
> When running {{hadoop fs -put bar swift://... }}, {{FsShell}} writes to
> {{bar._COPYING_}} first, and then renames it to "bar".
> However, it fails when the file is partitioned uploaded.
> {code}
> 15/02/06 21:00:49 DEBUG snative.SwiftNativeFileSystemStore: Source file
> appears to be partitioned. copying file and deleting children
> 15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter
> http.method.retry-handler =
> org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@26bf4e60
> 15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter
> http.connection.timeout = 15000
> 15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter
> http.socket.timeout = 60000
> 15/02/06 21:00:49 DEBUG http.SwiftRestClient: COPY
> https://snet-storage101.iad3.clouddrive.com/v1/MossoCloudFS_946265/foo/bar._COPYING_
> X-Auth-Token: ...
> Destination: foo/bar
> User-Agent: Apache Hadoop Swift Client 3.0.0-SNAPSHOT from ... by root source
> checksum ...
> 15/02/06 21:00:49 DEBUG httpclient.HttpConnection: Open connection to
> snet-storage101.iad3.clouddrive.com:443
> 15/02/06 21:00:49 DEBUG wire.header: >> "COPY
> /v1/MossoCloudFS_946265/foo/bar._COPYING_ HTTP/1.1[\r][\n]"
> 15/02/06 21:00:49 DEBUG httpclient.HttpMethodBase: Adding Host request header
> 15/02/06 21:00:49 DEBUG wire.header: >> "X-Auth-Token: ...[\r][\n]"
> 15/02/06 21:00:49 DEBUG wire.header: >> "Destination: foo/bar[\r][\n]"
> 15/02/06 21:00:49 DEBUG wire.header: >> "User-Agent: Apache Hadoop Swift
> Client 3.0.0-SNAPSHOT from ... by root source checksum ...[\r][\n]"
> 15/02/06 21:00:49 DEBUG wire.header: >> "Host:
> snet-storage101.iad3.clouddrive.com[\r][\n]"
> 15/02/06 21:00:49 DEBUG wire.header: >> "Content-Length: 0[\r][\n]"
> 15/02/06 21:00:49 DEBUG wire.header: >> "[\r][\n]"
> 15/02/06 21:00:49 DEBUG methods.EntityEnclosingMethod: Request body has not
> been specified
> 15/02/06 21:01:49 DEBUG httpclient.HttpMethodDirector: Closing the connection.
> 15/02/06 21:01:49 DEBUG httpclient.HttpMethodDirector: Method retry handler
> returned false. Automatic recovery will not be attempted
> 15/02/06 21:01:49 DEBUG httpclient.HttpConnection: Releasing connection back
> to connection manager.
> 15/02/06 21:01:49 DEBUG httpclient.HttpConnection: Releasing connection back
> to connection manager.
> {code}
> The {{bar}} file eventually shows up in Swift. However, the {{bar._COPYING_}}
> directories and the manifest file are not deleted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
