In now's hdfs implementation,all INodeFile and INodeDirectory objects were loaded into memory,this is done when setting up the FSNameSpacs structure set up at namenode startup. the namenode will analyze the fsimage file and edit log file. And if there are milllions of files or directories how it can be handled?
I have done an exprements by making dirs,before i exprements: [EMAIL PROTECTED] bin]$ ps -p 9122 -o rss,size,vsize,%mem RSS SZ VSZ %MEM 153648 1193868 1275340 3.7 after i creating 10000 directories, it turns: [EMAIL PROTECTED] bin]$ ps -p 9122 -o rss,size,vsize,%mem RSS SZ VSZ %MEM 169084 1193868 1275340 4.0 I m trying to improve the fsimage file,so that namenode can locate and load the needed information at need,and just like linux vfs,we can only obtain an inode cache.So this can avoid loading the whole namespace structure at startup. Best regards, Yongqiang He 2008-07-01 Email: [EMAIL PROTECTED] Tel: 86-10-62600966(O) Research Center for Grid and Service Computing, Institute of Computing Technology, Chinese Academy of Sciences P.O.Box 2704, 100080, Beijing, China
