On Jul 6, 2006, at 12:02 PM, Paul Sutter wrote:
...
*Constant size file blocks (#16), -1*
I vote to keep variable size blocks, especially because you are adding
atomic append capabilities (#25). Variable length blocks creates the
possibility for blocks that contain only whole records. This:
- improves recoverability for large important files with one or more
irrevocably lost blocks, and
- makes it very clean for mappers to process local data blocks
... I think we can achieve our goal without compromising yours.
Each block can be of any size up to the files fixed block size. The
system can be aware of that and provide an API to report gaps and/or
an API option to skip them or see them as NULLs. This reporting can
be done at the datanode level allowing us to remove all the size data
& logic at the namenode level.
** If you agree, why don't we just add the above annotation to
konstantine's doc?
*Recoverability and Availability Goals*
...
**
*Backup Scheme*
**
We might want to start discussion of a backup scheme for HDFS,
especially
given all the courageous rewriting and feature-addition likely to
occur.
** I agree, this needs to be on the list. I'm imagining a command
that hardlinks every datanode's (and namenode's if needed) files into
a snapshot directory. And another command that moves all current
state into a snapshot directory and hardlinks a snapshot's state back
into the working directory. This would be very fast and not cost
much space in the short term. Thoughts? (yes, hardlinks are a pain
on the PC, we can discuss design later)
*Rebalancing (#22,#21)*
I would suggest that keeping disk usage balanced is more than a
performance
feature, its important for the success of running jobs with large map
outputs or large sorts. Our most common reducer failure is running
out of
disk space during sort, and this is caused by imbalanced block
allocation.
** Good point. Any interest in helping us with this one?
On 6/30/06, Konstantin Shvachko <[EMAIL PROTECTED]> wrote:
I've created a Wiki page that summarizes DFS requirements and
proposed
changes.
This is a summary of discussions held in this mailing list and
additional internal discussions.
The page is here:
http://wiki.apache.org/lucene-hadoop/DFS_requirements
I see there is an ongoing related discussion in HADOOP-337.
We prioritized our goals as
(1) Reliability (which includes Recoverability and Availability)
(2) Scalability
(3) Functionality
(4) Performance
(5) other
But then gave higher priority to some features like the append
functionality.
Happy holidays to everybody.
--Konstantin Shvachko
