Lei (Eddy) Xu created HADOOP-11556:
--------------------------------------
Summary: SwiftNativeFileSystem rename being interrupted during "fs
-put"
Key: HADOOP-11556
URL: https://issues.apache.org/jira/browse/HADOOP-11556
Project: Hadoop Common
Issue Type: Bug
Components: fs, fs/swift
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
When running {{hadoop fs -put bar swift://... }}, {{FsShell}} writes to
{{bar._COPYING_}} first, and then renames it to "bar".
However, it fails when the file is partitioned uploaded.
{code}
15/02/06 21:00:49 DEBUG snative.SwiftNativeFileSystemStore: Source file appears
to be partitioned. copying file and deleting children
15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter
http.method.retry-handler =
org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@26bf4e60
15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter
http.connection.timeout = 15000
15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter
http.socket.timeout = 60000
15/02/06 21:00:49 DEBUG http.SwiftRestClient: COPY
https://snet-storage101.iad3.clouddrive.com/v1/MossoCloudFS_946265/foo/bar._COPYING_
X-Auth-Token: ...
Destination: foo/bar
User-Agent: Apache Hadoop Swift Client 3.0.0-SNAPSHOT from ... by root source
checksum ...
15/02/06 21:00:49 DEBUG httpclient.HttpConnection: Open connection to
snet-storage101.iad3.clouddrive.com:443
15/02/06 21:00:49 DEBUG wire.header: >> "COPY
/v1/MossoCloudFS_946265/foo/bar._COPYING_ HTTP/1.1[\r][\n]"
15/02/06 21:00:49 DEBUG httpclient.HttpMethodBase: Adding Host request header
15/02/06 21:00:49 DEBUG wire.header: >> "X-Auth-Token: ...[\r][\n]"
15/02/06 21:00:49 DEBUG wire.header: >> "Destination: foo/bar[\r][\n]"
15/02/06 21:00:49 DEBUG wire.header: >> "User-Agent: Apache Hadoop Swift Client
3.0.0-SNAPSHOT from ... by root source checksum ...[\r][\n]"
15/02/06 21:00:49 DEBUG wire.header: >> "Host:
snet-storage101.iad3.clouddrive.com[\r][\n]"
15/02/06 21:00:49 DEBUG wire.header: >> "Content-Length: 0[\r][\n]"
15/02/06 21:00:49 DEBUG wire.header: >> "[\r][\n]"
15/02/06 21:00:49 DEBUG methods.EntityEnclosingMethod: Request body has not
been specified
15/02/06 21:01:49 DEBUG httpclient.HttpMethodDirector: Closing the connection.
15/02/06 21:01:49 DEBUG httpclient.HttpMethodDirector: Method retry handler
returned false. Automatic recovery will not be attempted
15/02/06 21:01:49 DEBUG httpclient.HttpConnection: Releasing connection back to
connection manager.
15/02/06 21:01:49 DEBUG httpclient.HttpConnection: Releasing connection back to
connection manager.
{code}
The {{bar}} file eventually shows up in Swift. However, the {{bar._COPYING_}}
directories and the manifest file are not deleted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
