[
https://issues.apache.org/jira/browse/HADOOP-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12524010
]
Sameer Paranjpye commented on HADOOP-1700:
------------------------------------------
> One cost of this approach is that block revision numbers would consume more
> memory per block on the namenode, something we wish to minimize. So there is
> something of a time/space tradeoff.
We're saving hundreds of bytes with HADOOP-1687, what's the use of doing that
if we don't use them :)
But seriously, the space cost is very small, an additional 4-8 bytes/block. In
all of the reduction in memory usage proposed in HADOOP-1687, there has been
very little culling of file and block metadata. Most of the savings come from
the use of better containers, sorted lists vs treemaps, for instance and from
limiting the total number of Java objects the Namenode has in memory. My point
is that the data members of file and block objects have not been the source of
our memory problems and are unlikely to be so even if we add a few dozen bytes
to them.
> Thinking a bit more, it seems to me that with the new block-id approach, one
> could also manage things so that [ ... ]
That could be certainly be done.
One issue with new block ids though is that if a datanode fails in the middle
of an update, it would have both the old and new blocks, say <oldid> and
<newid>. Replicas where the update succeeded would remove the old block and
inform the Namenode, which would remove it from it's blocks map. The old block
would remain on the failed datanode and be orphaned. If the Namenode then
assigned this <oldid> to a new file, there would be a block-id collision. This
problem already exists to some degree. If a file is deleted and a datanode
containing that files blocks fails then these orphans remain on the failed node
and a collision can happen when it comes back.
If we use a revision number that takes the form of a timestamp, it can be used
to distinguish not only out of date replicas of a currently existing file but
also those from old long deleted files. Two birds and all that ...
> Append to files in HDFS
> -----------------------
>
> Key: HADOOP-1700
> URL: https://issues.apache.org/jira/browse/HADOOP-1700
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: stack
>
> Request for being able to append to files in HDFS has been raised a couple of
> times on the list of late. For one example, see
> http://www.nabble.com/HDFS%2C-appending-writes-status-tf3848237.html#a10916193.
> Other mail describes folks' workarounds because this feature is lacking:
> e.g. http://www.nabble.com/Loading-data-into-HDFS-tf4200003.html#a12039480
> (Later on this thread, Jim Kellerman re-raises the HBase need of this
> feature). HADOOP-337 'DFS files should be appendable' makes mention of file
> append but it was opened early in the life of HDFS when the focus was more on
> implementing the basics rather than adding new features. Interest fizzled.
> Because HADOOP-337 is also a bit of a grab-bag -- it includes truncation and
> being able to concurrently read/write -- rather than try and breathe new life
> into HADOOP-337, instead, here is a new issue focused on file append.
> Ultimately, being able to do as the google GFS paper describes -- having
> multiple concurrent clients making 'Atomic Record Append' to a single file
> would be sweet but at least for a first cut at this feature, IMO, a single
> client appending to a single HDFS file letting the application manage the
> access would be sufficent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
