i have to find Counter and i am using inetsocketAddress and connecting into the jobtracker jsp with help of it am able to find the counter when it is running on pseudo mode
but as i am trying to run this in cluster . i am not able to get any counters . SI there any parameters that I need to change in configuration to read the counter information ? Thanks in advance On Tue, Feb 9, 2010 at 8:49 PM, Jeff Zhang <[email protected]> wrote: > JobClient also use proxy of JobTracker. > > > On Mon, Feb 8, 2010 at 11:19 PM, Mark N <[email protected]> wrote: > > > Did you check the jobClient source code? > > > > > > On Thu, Feb 4, 2010 at 5:21 PM, Jeff Zhang <[email protected]> wrote: > > > > > I look at the source code, it seems the job tracker web ui also use the > > > proxy of JobTracker to get the counter information rather the xml file. > > > > > > > > > On Thu, Feb 4, 2010 at 7:29 PM, Mark N <[email protected]> wrote: > > > > > > > yes we can create a webservice in java which would be called by .net > to > > > > display these counters. > > > > > > > > But since the java code to read these counters needs use hadoop APIs > ( > > > job > > > > client ) , am not sure we can create a webservice to read the > > counters > > > > > > > > Question is how does the default hadoop task tracker display counter > > > > information in JSP pages ? does it read from the XML files ? > > > > > > > > thanks, > > > > > > > > On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <[email protected]> wrote: > > > > > > > > > I think you can create web service using Java, and then in .net > using > > > the > > > > > web service to display the result. > > > > > > > > > > > > > > > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[email protected]> > wrote: > > > > > > > > > > > Do you mean want to connect the JobTracker using .Net ? If so, > I'm > > > > afraid > > > > > I > > > > > > have no idea how to this. The rpc of hadoop is language > dependent. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[email protected]> > > wrote: > > > > > > > > > > > >> could you please elaborate on this ( * hint to get started as > am > > > > very > > > > > >> new > > > > > >> to hadoop? ) > > > > > >> So far I could succesfully read all the default and custom > > counters. > > > > > >> > > > > > >> Currently we are having a .net client. > > > > > >> > > > > > >> thanks in advance. > > > > > >> > > > > > >> > > > > > >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[email protected]> > > > wrote: > > > > > >> > > > > > >> > Well, you can create a proxy of JobTracker in client side, and > > > then > > > > > you > > > > > >> can > > > > > >> > use the API of JobTracker to get the information of jobs. The > > > Proxy > > > > > take > > > > > >> > the > > > > > >> > responsibility of communication with the Master Node. Read > the > > > > > source > > > > > >> > code > > > > > >> > of JobClient can help you. > > > > > >> > > > > > > >> > > > > > > >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[email protected]> > > > > wrote: > > > > > >> > > > > > > >> > > Ye currently am using jobclient to read these counters. > > > > > >> > > > > > > > >> > > But We are not able to use *webservices *because the jar > which > > > is > > > > > used > > > > > >> to > > > > > >> > > read the counters from running hadoop job is itself a > Hadoop > > > > > program > > > > > >> > > > > > > > >> > > If we could have pure Java Api which is run without hadoop > > > command > > > > > >> then > > > > > >> > we > > > > > >> > > could return the counter variable into webservices and show > in > > > UI. > > > > > >> > > > > > > > >> > > Any help or technique to show thsese counters in the UI > would > > > be > > > > > >> > > appreciated ( not necessarily using web service ) > > > > > >> > > > > > > > >> > > > > > > > >> > > I am using webservices because I am having .net VB client > > > > > >> > > > > > > > >> > > thanks > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang < > [email protected]> > > > > > wrote: > > > > > >> > > > > > > > >> > > > I think you can use JobClient to get the counters in your > > web > > > > > >> service. > > > > > >> > > > If you look at the shell script bin/hadoop, you will find > > that > > > > > >> actually > > > > > >> > > > this > > > > > >> > > > shell use the JobClient to get the counters. > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N < > > [email protected]> > > > > > >> wrote: > > > > > >> > > > > > > > > >> > > > > We have a hadoop job running and have used custom > counters > > > to > > > > > >> track > > > > > >> > > few > > > > > >> > > > > counters ( like no of successfully processed documents > > > > matching > > > > > >> > certain > > > > > >> > > > > conditions) > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > Since we need to get this counters even while the Hadoop > > job > > > > is > > > > > >> > running > > > > > >> > > , > > > > > >> > > > > we > > > > > >> > > > > wrote another Java program to read these counters > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > *Counter reader program *will do the following : > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > 1) List all the running jobs. > > > > > >> > > > > > > > > > >> > > > > 2) Get the running job using Job name > > > > > >> > > > > > > > > > >> > > > > 2) Get all the counter for individual running jobs > > > > > >> > > > > > > > > > >> > > > > 3) Set this counters in variables. > > > > > >> > > > > We could successfully read these counters , but > > > since > > > > we > > > > > >> need > > > > > >> > > to > > > > > >> > > > > show these counters to custom UI , how can we show these > > > > > counters? > > > > > >> > > > > > > > > > >> > > > > we looked into various options to read these > > counters > > > > to > > > > > >> show > > > > > >> > in > > > > > >> > > > UI > > > > > >> > > > > as following : > > > > > >> > > > > > > > > > >> > > > > 1. Dump these counters to database , however this > may > > > be > > > > > >> > overhead > > > > > >> > > > > 2. Write web service and UI will invoke the > > functions > > > > > from > > > > > >> > these > > > > > >> > > > > service to show in UI ( However since we need to run > > > "*Counter > > > > > >> reader > > > > > >> > > > > program " *with Hadoop command it might not be feasible > > to > > > > > write > > > > > >> web > > > > > >> > > > > service ? ) > > > > > >> > > > > > > > > > >> > > > > so the question is can we achive to read the > counters > > > > using > > > > > >> > simple > > > > > >> > > > > Java APIs ? Does anyone have idea how does the default > > > > > jobtracker > > > > > >> JSP > > > > > >> > > > works > > > > > >> > > > > ? we wanted to built something similar to this > > > > > >> > > > > > > > > > >> > > > > thanks > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > -- > > > > > >> > > > > Nipen Mark > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > -- > > > > > >> > > > Best Regards > > > > > >> > > > > > > > > >> > > > Jeff Zhang > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > -- > > > > > >> > > Nipen Mark > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > -- > > > > > >> > Best Regards > > > > > >> > > > > > > >> > Jeff Zhang > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> Nipen Mark > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best Regards > > > > > > > > > > > > Jeff Zhang > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best Regards > > > > > > > > > > Jeff Zhang > > > > > > > > > > > > > > > > > > > > > -- > > > > Nipen Mark > > > > > > > > > > > > > > > > -- > > > Best Regards > > > > > > Jeff Zhang > > > > > > > > > > > -- > > Nipen Mark > > > > > > -- > Best Regards > > Jeff Zhang > -- Nipen Mark
