For Req1, a better approach might be to have a single "error reporting" operator that connects to the DB; all the other operators can send custom error tuples to it with appropriate details of the type of error. That way, if there are issues that are DB-related, you have 1 place to debug them instead of 20.
With that approach, your DB operator will need 20 input ports and a new one added each time you add an additional operator to the DAG; an alternative is to use the OperatorRequest mechanism to communicate with the DB operator instead of the normal ports-and-tuples method. This is illustrated in the example at https://github.com/DataTorrent/examples/tree/master/tutorials/throttle where it is used to modulate the output data rate of the input operator when the downstream operators are unable to keep up. For Req2, you can put the DB properties in an HDFS location and read it in the setup() callback of the operator. The path to that location can be in your regular properties file within the app package. A similar, though more elaborate method is to run a service on a known port which can be queried for those properties; the operators will then connect to that service in setup() to retrieve the info. For Req3, cluster-level issues are generally not made available to user-level callbacks; you'll need an external process(es) to monitor log files, invoke REST APIs etc. Ram On Thu, Nov 17, 2016 at 5:50 AM, Venky <[email protected]> wrote: > Hi, > > Req1: I have around 20 operators. In every operator I want to connect to > database to mark my processing as failed when there is an Io Exception or > any cluster level exception. I wanted to use plain java class here, please > let me know how we can implement it. If not suggest the better approach > > Req2: I don't want the DB properties file within the application package > and wanted to maintain the DB properties outside in a properties file and > able to use them while connecting to database to fulfill my Req1. Please > let me know how we can maintain the DB properties file outside. > > Req3: How can I capture cluster level exceptions like RPC Failure and etc? > > Regards, > Venkat. >
