Hello all, I have completed creating a stand-alone implementation/POC of job monitor in the monitoring module of the codebase.
I did fork from a dev branch and code, but I was unable to commit. I tried creating patch, but the patch did not aggregate all the changes, so I went ahead and just created a copy of the module (I think it should be as good as a patch) for now and attached it to the JIRA issues: https://issues.apache.org/jira/browse/AIRAVATA-1912 https://issues.apache.org/jira/browse/AIRAVATA-1914 https://issues.apache.org/jira/browse/AIRAVATA-1915 This code addresses the sub-task: · AIRAVATA-1914 · AIRAVATA-1915 I have also flagged the above two sub-tasks as patch. Briefly this is what stand-alone implementation does right now: · Creates a fanout exchange with durable queues (will persist after server restarts) and auto-recovery option enabled; declares and binds all the consumer queues · Fetch e-mails from a dummy email address · Converts the *non-serializable* javax Message objects into custom *serializable* MessageExtract objects, which consists of only the necessary information, viz. “from”, “to”, “subject” and “content” fields, which are being used in the existing codebase · Serialize the messages · Send the message to the exchange, which in turn will deliver the message to all the queues bound to it · Two consumers threads fetch the messages from their respective queues · Deserialize the message and convert it into javax Message object, so that it can be used with the existing code · Print the received message content I have tried to keep all the components object-oriented. Following is the explanation of the components, Core Components: · RabbitMQEmailPublisher: RabbitMQ publisher for e-mail messages · EmailReceiver: Represents a consumer, which spawns a thread and keeps on receiving messages and process them. Multiple EmailReceiver can be instantiated and they will automatically run in different threads. · GmailSMTPMailBox: represents gmail mail box · EmailConsumer: This has the implementation of call back for received messages. · MessageExtract: Represents the extract from e-mail, which is serializable Simulation Components: · FetchPublish: Fetches emails with GmailSMTPMailBox and publishes to message bus with RabbitMQEmailPublisher · Simulator (also the main class to run): o Instantiates two consumers (EmailReceiver) in separate threads o Fetches email and publishes it with FetchPublish o Shuts down the consumers after a minute Following is the TO-DO list: · Use the actual email server configs o Currently I am using my own dummy account for testing, did not want to accidentally mess up the production mail-boxes · Use the codebase broker config o Currently I am using a local broker url and config · Fetch the actual email text/body o Currently I am using a hard-coded string for email text, as somehow when I extract the content, I always get multi-part content. The code you guys already have in the repository seems to be written with an assumption that the received message content will be text only, I tried creating an email like that, but it always has multi-part component. So once I have integrated the code with the existing monitor, I can get back to fetching the actual content · Open the inbox in read_write mode o Currently I am opening the inbox in read mode, so that the messages are not marked as read and so I don’t have to repeatedly create new dummy messages · Probably implement a RabbitMQ Factory singleton · Disable message(from message queue) auto-ack and only ack after processing has been completed, this might help making the workflow more robust · Use the codebase properties file for configuration of broker and email service, right now creating properties on run time · Create test-cases I looked into integrating this into the live work-flow, but following things kind of need time: · The message processing work-flow will have to be inverted, it’s kind of different than we assumed. Currently the messages are fetched explicitly and then processing is done. The way I have implemented the consumers is, the queue messages will be delivered asynchronously, so we don’t have to poll for it. So all the processing part will have to be wired with call-back function. · Setup PGA and its dependencies on local for testing · Understand in which part of the life-cycle the consumer(or equivalent of that) threads are to be invocated and what specific shutdown interrupts are they to follow and respond to · Understand properties files conventions Hence the aforementioned list with the TO-DO list will be part of milestone 1. Best regards, Siddharth Jain
