About req 3: AFAIK CDH does provide REST access to cluster health parameters among many other things. Not sure about ambari though
Thanks, A _____________________________________ Sent with difficulty, I mean handheld ;) On 17 Nov 2016 8:25 pm, "Munagala Ramanath" <[email protected]> wrote: > For Req1, a better approach might be to have a single "error reporting" > operator that connects to the DB; > all the other operators can send custom error tuples to it with > appropriate details of the type of error. > That way, if there are issues that are DB-related, you have 1 place to > debug them instead of 20. > > With that approach, your DB operator will need 20 input ports and a new > one added each time you add > an additional operator to the DAG; an alternative is to use the > OperatorRequest mechanism to > communicate with the DB operator instead of the normal ports-and-tuples > method. This is illustrated > in the example at https://github.com/DataTorrent/examples/tree/ > master/tutorials/throttle > where it is used to modulate the output data rate of the input operator > when the downstream operators > are unable to keep up. > > For Req2, you can put the DB properties in an HDFS location and read it in > the setup() callback of the > operator. The path to that location can be in your regular properties file > within the app package. > A similar, though more elaborate method is to run a service on a known > port which can be queried > for those properties; the operators will then connect to that service in > setup() to retrieve the info. > > For Req3, cluster-level issues are generally not made available to > user-level callbacks; you'll need > an external process(es) to monitor log files, invoke REST APIs etc. > > Ram > > > > On Thu, Nov 17, 2016 at 5:50 AM, Venky <[email protected]> wrote: > >> Hi, >> >> Req1: I have around 20 operators. In every operator I want to connect to >> database to mark my processing as failed when there is an Io Exception or >> any cluster level exception. I wanted to use plain java class here, please >> let me know how we can implement it. If not suggest the better approach >> >> Req2: I don't want the DB properties file within the application package >> and wanted to maintain the DB properties outside in a properties file and >> able to use them while connecting to database to fulfill my Req1. Please >> let me know how we can maintain the DB properties file outside. >> >> Req3: How can I capture cluster level exceptions like RPC Failure and etc? >> >> Regards, >> Venkat. >> > >
