[
http://issues.apache.org/jira/browse/HADOOP-224?page=comments#action_12413020 ]
Konstantin Shvachko commented on HADOOP-224:
--------------------------------------------
I thought that we might want to use java reflections to make versioning
support more generic.
This would require some programming discipline and the reflection framework
will do the rest.
So I suppose that all classes that require versioning implement Versioned (let
me know if the
name does not sound right) interface that should include getVersion() method.
Additionally all versions of the same class should implement the same
interface, which declares
methods that are version dependent.
For example lets consider the INode class.
public class INode implements INodeReader, Versioned {
String name;
int nrBlocks;
....
}
INodeReader is the interface that declares e.g. only one method readFields( in )
and each version of INode should implement this interface.
For each field declared in the INode class we must have corresponding methods
get<fieldName>
set<fieldName>
setDefault<fieldName>
For now I assume that all fields are of primitive types, and that version
transition
means adding or removing fields in the class only.
Then we should have a procedure of retiring old versions, which I see as
renaming
the package of the Versioned class such that it includes version number. E.g.
org.apache.hadoop.dfs.INode
is renamed to
org.apache.hadoop.dfs.v0.INode
if the old version is 0 and the new one is 1.
The retired classes are placed in a separate jar file.
I didn't think whether the retiring can be automated with an ant script or not.
Finally we have VersionFactory class, which can be either
a member or a superclass of INode.
The implementation of readFields is simple:
INode.readFields( in ) {
int storedVersion = in.readVersion();
VersionFactory.read( getCurrentVersion(), storedVersion, this );
}
And then VersionFactory.read() does the actual job.
VersionFactory.read( targetVersion, sourceVersion, targetClass ) {
get class name of the required version base on targetClass name and
sourceVersion;
construct the sourceClass;
if class not found report return that the version is not supported;
targetFields = targetClass.getFields() and sort them lexicographically by
field name;
sourceFields = sourceClass.getFields() and sort them lexicographically by
field name;
Then we scan the two lists.
If both of them contain field A (of type T) then {
// this common field of the two versions
T value = in.readT();
and then invoke targetClass.setA( value )
}
If field A is contained only in targetClass {
// this is a new field
invoke targetClass.setDefaultA();
}
if field A belongs only to sourceClass {
// this field was removed in the new version
T value = in.readT();
and do not assign it to anything;
}
}
Advantages:
That way we can read data from any previous version not only the preceding one.
And when defining a new version of the class we do not need to have any
knowledge
of the previous version(s).
Also we can and should have many Versioned classes (for INode, for add, delete,
rename operation logs, ...) but we can use the same VersionFactory for all of
them.
> Allow simplified versioning for namenode and datanode metadata.
> ---------------------------------------------------------------
>
> Key: HADOOP-224
> URL: http://issues.apache.org/jira/browse/HADOOP-224
> Project: Hadoop
> Type: Improvement
> Components: dfs
> Environment: All
> Reporter: Milind Bhandarkar
>
> Currently namenode has two types of metadata: The FSImage, and FSEdits.
> FSImage contains information abut Inodes, and FSEdits contains a list of
> operations that were not saved to FSImage. Datanode currently does not have
> any metadata, but would have it some day.
> The file formats used for storing these metadata will evolve over time. It is
> important for the file-system to be backward compatible. That is, the
> metadata readers need to be able to identify which version of the file-format
> we are using, and need to be able to read information therein. As we add
> information to these metadata, the complexity of the reader increases
> dramatically.
> I propose a versioning scheme with a major and minor version number, where a
> different reader class is associated with a major number, and that class
> interprets the minor number internally. The readers essentially form a chain
> starting with the latest version. Each version-reader looks at the file and
> if it does not recognize the version number, passes it to the version reader
> next to it by calling the parse method, returnng the results of the parse
> method up the chain (In case of the namenode, the parse result is an array of
> Inodes.
> This scheme has an advantage that every time a new major version is added,
> the new reader only needs to know about the reader for its immediately
> previous version, and every reader needs to know only about which major
> version numbers it can read.
> The writer is not so versioned, because metadata is always written in the
> most current version format.
> One more change that is needed for simplified versioning is that the
> "struct-surping" of dfs.Block needs to be removed. Block's contents will
> change in later versions, and older versions should still be able to
> readFields properly. This is more general than Block of course, and in
> general only basic datatypes should be used as Writables in DFS metadata.
> For edits, the reader should return <opcode, ArrayWritable> pairs' array.
> This will also remove the limitation of two operands for very opcodes, and
> will be more extensible.
> Even with this new versioning scheme, the last Reader in the reader-chain
> would recognize current format, thus maintaining full backward compatibility.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
