[email protected] wrote on 10/18/11 at 09:23:50 -0700: >Cool, is there any documentation on how to use the JMX stuff to get >monitoring data?
I don't know if there is any specific documentation. These are the mbeans you might be interested in Namenode: Hadoop:service=NameNode,name=FSNamesystemState Hadoop:service=NameNode,name=NameNodeInfo Hadoop:service=NameNode,name=jvm JobTracker: Hadoop:service=JobTracker,name=JobTrackerInfo Hadoop:service=JobTracker,name=QueueMetrics,q=<queuename> Hadoop:service=JobTracker,name=jvm DataNode: Hadoop:name=DataNodeInfo,service=DataNode TaskTracker: Hadoop:service=TaskTracker,name=TaskTrackerInfo You may also want to monitor shuffle_exceptions_caught in Hadoop:service=TaskTracker,name=ShuffleServerMetrics > >Cheers, >Ivan > >On 10/17/11 6:04 PM, "Rajiv Chittajallu" <[email protected]> wrote: > >>If you are running > 0.20.204 >>http://phanpy-nn1.hadoop.apache.org:50070/jmx?qry=Hadoop:service=NameNode, >>name=NameNodeInfo >> >> >>[email protected] wrote on 10/17/11 at 09:18:20 -0700: >>>Hi Harsh, >>> >>>I need access to the data programatically for system automation, and >>>hence >>>I do not want a monitoring tool but access to the raw data. >>> >>>I am more than happy to use an exposed function or client program and not >>>an internal API. >>> >>>So i am still a bit confused... What is the simplest way to get at this >>>raw disk usage data programmatically? Is there a HDFS equivalent of du >>>and df, or are you suggesting to just run that on the linux OS (which is >>>perfectly doable). >>> >>>Cheers, >>>Ivan >>> >>> >>>On 10/17/11 9:05 AM, "Harsh J" <[email protected]> wrote: >>> >>>>Uma/Ivan, >>>> >>>>The DistributedFileSystem class explicitly is _not_ meant for public >>>>consumption, it is an internal one. Additionally, that method has been >>>>deprecated. >>>> >>>>What you need is FileSystem#getStatus() if you want the summarized >>>>report via code. >>>> >>>>A job, that possibly runs "du" or "df", is a good idea if you >>>>guarantee perfect homogeneity of path names in your cluster. >>>> >>>>But I wonder, why won't using a general monitoring tool (such as >>>>nagios) for this purpose cut it? What's the end goal here? >>>> >>>>P.s. I'd moved this conversation to hdfs-user@ earlier on, but now I >>>>see it being cross posted into mr-user, common-user, and common-dev -- >>>>Why? >>>> >>>>On Mon, Oct 17, 2011 at 9:25 PM, Uma Maheswara Rao G 72686 >>>><[email protected]> wrote: >>>>> We can write the simple program and you can call this API. >>>>> >>>>> Make sure Hadoop jars presents in your class path. >>>>> Just for more clarification, DN will send their stats as parts of >>>>>hertbeats, So, NN will maintain all the statistics about the diskspace >>>>>usage for the complete filesystem and etc... This api will give you >>>>>that >>>>>stats. >>>>> >>>>> Regards, >>>>> Uma >>>>> >>>>> ----- Original Message ----- >>>>> From: [email protected] >>>>> Date: Monday, October 17, 2011 9:07 pm >>>>> Subject: Re: Is there a good way to see how full hdfs is >>>>> To: [email protected], [email protected] >>>>> Cc: [email protected] >>>>> >>>>>> So is there a client program to call this? >>>>>> >>>>>> Can one write their own simple client to call this method from all >>>>>> diskson the cluster? >>>>>> >>>>>> How about a map reduce job to collect from all disks on the cluster? >>>>>> >>>>>> On 10/15/11 4:51 AM, "Uma Maheswara Rao G 72686" >>>>>> <[email protected]>wrote: >>>>>> >>>>>> >/** Return the disk usage of the filesystem, including total >>>>>> capacity,> * used space, and remaining space */ >>>>>> > public DiskStatus getDiskStatus() throws IOException { >>>>>> > return dfs.getDiskStatus(); >>>>>> > } >>>>>> > >>>>>> >DistributedFileSystem has the above API from java API side. >>>>>> > >>>>>> >Regards, >>>>>> >Uma >>>>>> > >>>>>> >----- Original Message ----- >>>>>> >From: wd <[email protected]> >>>>>> >Date: Saturday, October 15, 2011 4:16 pm >>>>>> >Subject: Re: Is there a good way to see how full hdfs is >>>>>> >To: [email protected] >>>>>> > >>>>>> >> hadoop dfsadmin -report >>>>>> >> >>>>>> >> On Sat, Oct 15, 2011 at 8:16 AM, Steve Lewis >>>>>> >> <[email protected]> wrote: >>>>>> >> > We have a small cluster with HDFS running on only 8 nodes - I >>>>>> >> believe that >>>>>> >> > the partition assigned to hdfs might be getting full and >>>>>> >> > wonder if the web tools or java api havew a way to look at free >>>>>> >> space on >>>>>> >> > hdfs >>>>>> >> > >>>>>> >> > -- >>>>>> >> > Steven M. Lewis PhD >>>>>> >> > 4221 105th Ave NE >>>>>> >> > Kirkland, WA 98033 >>>>>> >> > 206-384-1340 (cell) >>>>>> >> > Skype lordjoe_com >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> >>>>>> > >>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>>>-- >>>>Harsh J >>>> >>> >
pgpxa4Xb50vLL.pgp
Description: PGP signature
