Hi Mich, the book Hadoop Operations may a good start: https://books.google.de/books?id=drbI_aro20oC&pg=PA308&lpg=PA308&dq=hadoop+memory+namenode&source=bl&ots=t_yltgk_i7&sig=_6LXkcSjfuwwqfz_kDGDi9ytgqU&hl=en&sa=X&ei=Nt8TVfn9AcjLPZyXgKAC&ved=0CFYQ6AEwBg#v=onepage&q=hadoop%20memory%20namenode&f=false <https://books.google.de/books?id=drbI_aro20oC&pg=PA308&lpg=PA308&dq=hadoop+memory+namenode&source=bl&ots=t_yltgk_i7&sig=_6LXkcSjfuwwqfz_kDGDi9ytgqU&hl=en&sa=X&ei=Nt8TVfn9AcjLPZyXgKAC&ved=0CFYQ6AEwBg#v=onepage&q=hadoop memory namenode&f=false>
BR, AL > On 26 Mar 2015, at 11:16, Mich Talebzadeh <[email protected]> wrote: > > Is there any parameter that sets the total memory that NameNode can use? > > Thanks > > Mich Talebzadeh > > http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> > > Publications due shortly: > Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and > Coherence Cache > > NOTE: The information in this email is proprietary and confidential. This > message is for the designated recipient only, if you are not the intended > recipient, you should destroy it immediately. Any information in this message > shall not be understood as given or endorsed by Peridale Ltd, its > subsidiaries or their employees, unless expressly so stated. It is the > responsibility of the recipient to ensure that this email is virus free, > therefore neither Peridale Ltd, its subsidiaries nor their employees accept > any responsibility. > > From: Mirko Kämpf [mailto:[email protected] > <mailto:[email protected]>] > Sent: 25 March 2015 16:08 > To: [email protected] <mailto:[email protected]>; > [email protected] <mailto:[email protected]> > Subject: Re: can block size for namenode be different from wdatanode block > size? > > Correct, let's say you run the NameNode with just 1GB of RAM. > This would be a very strong limitation for the cluster. For each file we need > about 200 bytes and for each block as well. Now we can estimate the max. > capacity depending on HDFS-Blocksize and average File size. > > Cheers, > Mirko > > 2015-03-25 15:34 GMT+00:00 Mich Talebzadeh <[email protected] > <mailto:[email protected]>>: > Hi Mirko, > > Thanks for feedback. > > Since i have worked with in memory databases, this metadata caching sounds > more like an IMDB that caches data at start up from disk resident storage. > > IMDBs tend to get issues when the cache cannot hold all data. Is this the > case the case with metada as well? > > Regards, > > Mich > Let your email find you with BlackBerry from Vodafone > From: Mirko Kämpf <[email protected] <mailto:[email protected]>> > Date: Wed, 25 Mar 2015 15:20:03 +0000 > To: [email protected] > <mailto:[email protected]><[email protected] > <mailto:[email protected]>> > ReplyTo: [email protected] <mailto:[email protected]> > Subject: Re: can block size for namenode be different from datanode block > size? > > Hi Mich, > > please see the comments in your text. > > > > 2015-03-25 15:11 GMT+00:00 Dr Mich Talebzadeh <[email protected] > <mailto:[email protected]>>: > > Hi, > > The block size for HDFS is currently set to 128MB by defauilt. This is > configurable. > Correct, an HDFS client can overwrite the cfg-property and define a different > block size for HDFS blocks. >> >> My point is that I assume this parameter in hadoop-core.xml sets the >> block size for both namenode and datanode. > Correct, the block-size is a "HDFS wide setting" but in general the > HDFS-client makes the blocks. > >> However, the storage and >> random access for metadata in nsamenode is different and suits smaller >> block sizes. > HDFS blocksize has no impact here. NameNode metadata is held in memory. For > reliability it is dumped to local discs of the server. > >> >> For example in Linux the OS block size is 4k which means one HTFS blopck >> size of 128MB can hold 32K OS blocks. For metadata this may not be >> useful and smaller block size will be suitable and hence my question. > Remember, metadata is in memory. The fsimage-file, which contains the > metadata > is loaded on startup of the NameNode. > > Please be not confused by the two types of block-sizes. > > Hope this helps a bit. > Cheers, > Mirko > >> >> Thanks, >> >> Mich
