You can use org.apache.hadoop.ipc.RPC.getProxy(....) to initialize the proxy of JobTracker
On Thu, Feb 4, 2010 at 7:23 PM, Jeff Zhang <[email protected]> wrote: > I think you can create web service using Java, and then in .net using the > web service to display the result. > > > > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[email protected]> wrote: > >> Do you mean want to connect the JobTracker using .Net ? If so, I'm afraid >> I have no idea how to this. The rpc of hadoop is language dependent. >> >> >> >> >> On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[email protected]> wrote: >> >>> could you please elaborate on this ( * hint to get started as am very >>> new >>> to hadoop? ) >>> So far I could succesfully read all the default and custom counters. >>> >>> Currently we are having a .net client. >>> >>> thanks in advance. >>> >>> >>> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[email protected]> wrote: >>> >>> > Well, you can create a proxy of JobTracker in client side, and then you >>> can >>> > use the API of JobTracker to get the information of jobs. The Proxy >>> take >>> > the >>> > responsibility of communication with the Master Node. Read the source >>> > code >>> > of JobClient can help you. >>> > >>> > >>> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[email protected]> wrote: >>> > >>> > > Ye currently am using jobclient to read these counters. >>> > > >>> > > But We are not able to use *webservices *because the jar which is >>> used to >>> > > read the counters from running hadoop job is itself a Hadoop >>> program >>> > > >>> > > If we could have pure Java Api which is run without hadoop command >>> then >>> > we >>> > > could return the counter variable into webservices and show in UI. >>> > > >>> > > Any help or technique to show thsese counters in the UI would be >>> > > appreciated ( not necessarily using web service ) >>> > > >>> > > >>> > > I am using webservices because I am having .net VB client >>> > > >>> > > thanks >>> > > >>> > > >>> > > >>> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[email protected]> wrote: >>> > > >>> > > > I think you can use JobClient to get the counters in your web >>> service. >>> > > > If you look at the shell script bin/hadoop, you will find that >>> actually >>> > > > this >>> > > > shell use the JobClient to get the counters. >>> > > > >>> > > > >>> > > > >>> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[email protected]> >>> wrote: >>> > > > >>> > > > > We have a hadoop job running and have used custom counters to >>> track >>> > > few >>> > > > > counters ( like no of successfully processed documents matching >>> > certain >>> > > > > conditions) >>> > > > > >>> > > > > >>> > > > > Since we need to get this counters even while the Hadoop job is >>> > running >>> > > , >>> > > > > we >>> > > > > wrote another Java program to read these counters >>> > > > > >>> > > > > >>> > > > > *Counter reader program *will do the following : >>> > > > > >>> > > > > >>> > > > > 1) List all the running jobs. >>> > > > > >>> > > > > 2) Get the running job using Job name >>> > > > > >>> > > > > 2) Get all the counter for individual running jobs >>> > > > > >>> > > > > 3) Set this counters in variables. >>> > > > > We could successfully read these counters , but since we >>> need >>> > > to >>> > > > > show these counters to custom UI , how can we show these >>> counters? >>> > > > > >>> > > > > we looked into various options to read these counters to >>> show >>> > in >>> > > > UI >>> > > > > as following : >>> > > > > >>> > > > > 1. Dump these counters to database , however this may be >>> > overhead >>> > > > > 2. Write web service and UI will invoke the functions from >>> > these >>> > > > > service to show in UI ( However since we need to run "*Counter >>> reader >>> > > > > program " *with Hadoop command it might not be feasible to write >>> web >>> > > > > service ? ) >>> > > > > >>> > > > > so the question is can we achive to read the counters using >>> > simple >>> > > > > Java APIs ? Does anyone have idea how does the default jobtracker >>> JSP >>> > > > works >>> > > > > ? we wanted to built something similar to this >>> > > > > >>> > > > > thanks >>> > > > > >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > Nipen Mark >>> > > > > >>> > > > >>> > > > >>> > > > >>> > > > -- >>> > > > Best Regards >>> > > > >>> > > > Jeff Zhang >>> > > > >>> > > >>> > > >>> > > >>> > > -- >>> > > Nipen Mark >>> > > >>> > >>> > >>> > >>> > -- >>> > Best Regards >>> > >>> > Jeff Zhang >>> > >>> >>> >>> >>> -- >>> Nipen Mark >>> >> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > > > -- > Best Regards > > Jeff Zhang > -- Best Regards Jeff Zhang
