Hi, Thanks for the feedback. I think that the FSNameSystem is the object used by NameNode to store metadata information, which are as straighforward as the ones used by the direct alternative to hadoop (PVFS2) [see 1] Though I think that being -Java- OOP, hadoop can serve higher level and more complex frameworks. My idea for metadata is not clear yet, but agrees with Mrs. Kate's viewpoint, hopefully to support for security/permissions, maybe consistency too, and why not some more advanced/experimental features. As a research subject, I am interested in the possibility of a metadata DBMS
Your suggestions/opinions are very much welcome. Yours, K. Honsali [1: PVFS2 metadata] from docs: *metadata*data about data. in the UNIX sense, such things as owner, group, permissions, timestamps, sizes. in the PVFS sense, also distribution information. On 27/09/2007, kate rhodes <[EMAIL PROTECTED] > wrote: > > I was assuming Khalil was speaking of more user specified meta-data > not your standard simple file/permissions meta-data that it would be > nigh-impossible to have a filesystem without. > > I could see great value in having each file contain info about the > process that generated, or a hint as to what it contains for mappers > to use. Without meta-data like that there's no way for a mapper to > differentiate one file in a folder from another.... the filesystem > becomes a defacto metadata store ... you have to describe/infer the > file's contents by where you put them. There's no way to inspect a > file's metadata to see if it's from a "foo" type MR or a "bar" type > MR. Does this file contain data about users or data about logs or data > about data? Only way to tell is where it was stuck in the filesystem. > > Wouldn't it be much better if you could just store some meta-data with > the file? Even if it was totally simplistic like e-mail headers... > > > > - kate = masukomi > http://weblog.masukomi.org/ > > > > On 9/26/07, Ted Dunning <[EMAIL PROTECTED]> wrote: > > > > The blocks in a file are stored on the namenode (clearly these are file > > meta-data). The namenode also stores the number of replications. > > > > There is a CRC for each block that is stored by the datanodes. > > > > Based on IRC chatter, there will soon be user and group permission meta > > data. > > > > > > On 9/26/07 2:01 PM, "kate rhodes" <[EMAIL PROTECTED]> wrote: > > > > > My understanding is that there is no file meta-data beyond it's name > > > and the directory that it lives in. > > > > > -- --------------------------------------------------------- Honsali Khalil − 本査理 カリル Academic>Japan>NIT>Grad. Sc. Eng.>Dept. CS>Hiroshi&Tsumura Lab. http://www.matlab.nitech.ac.jp/~k-hon/<http://www.matlab.nitech.ac.jp/%7Ek-hon/> +81 (zero-)eight-zero 5134 8119 [EMAIL PROTECTED] (instant reply mail)
