[
https://issues.apache.org/jira/browse/HADOOP-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263293#comment-15263293
]
Andres Perez commented on HADOOP-12057:
---------------------------------------
I used a Bluemix Object Store, where you don't have access to the swift.conf
file to change the default maximum object size limit. I think this is good
because it separate the dependency between needing to change something in swift
for Hadoop to work. This just uses the [Large Object Direct
API|http://docs.openstack.org/developer/swift/overview_large_objects.html#additional-notes]
of Swift, which in theory will allow to store 1TB files in several segments
without merging them, but still providing a single file to download and combine
all the pieces together.
The only issues if that is you hdfs dfs -ls swift://container.store/ you will
see the file displaying with size 0, which is also expected from the
documentation linked above.
> swiftfs rename on partitioned file attempts to consolidate partitions
> ---------------------------------------------------------------------
>
> Key: HADOOP-12057
> URL: https://issues.apache.org/jira/browse/HADOOP-12057
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/swift
> Reporter: David Dobbins
> Assignee: David Dobbins
> Attachments: HADOOP-12057-006.patch, HADOOP-12057-008.patch,
> HADOOP-12057.007.patch, HADOOP-12057.patch, HADOOP-12057.patch,
> HADOOP-12057.patch, HADOOP-12057.patch, HADOOP-12057.patch
>
>
> In the swift filesystem for openstack, a rename operation on a partitioned
> file uses the swift COPY operation, which attempts to consolidate all of the
> partitions into a single object. This causes the rename to fail when the
> total size of all the partitions exceeds the maximum object size for swift.
> Since partitioned files are primarily created to allow a file to exceed the
> maximum object size, this bug makes writing to swift extremely unreliable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
