On Sun, Apr 19, 2020 at 9:22 PM DImuthu Upeksha <[email protected]> wrote:
> Hi Dinuka, > > I will reply inline > > On Fri, Apr 17, 2020 at 12:37 PM Dinuka Desilva <[email protected]> > wrote: > >> Hi, >> >> I'm currently looking in to convert the existing java implementation of >> agent module to golang considering better concurrency, performance and make >> it more platform enabled. Before just starting up, I'm trying to >> discuss which kind of changes has to be done on which end. >> >> 1) I believe we are considering to accommodate multiple agents available >> per controller and each agent to support multiple transports. Which means, >> it's a collection of agents connected to a one controller. I think this is >> not done yet. >> >> There should be only one active controller for entire system. Controller > is the master and agents are the workers. This follows the master slave > architecture. Controller sends messages to agents when a transfer needs to > be performed. And this is already in place. > Got it. > > 2) Currently my understanding according to the implementation is that the >> agent module is kind of coupled with the api and controller. I feel that >> agent has to be an independent module. Or am I missing something? >> >> > Agent is not coupled with the controller. But it has a standard messaging > API built on top of Consul. If controller needs to get some work done by > the agent, it has to communicate in that particular messaging API. But > there is no code level dependency between the controller and agents. > However Agent has dependencies with Resource and Secret APIs to fetch > metadata of the resources. This is also not with the Service but with the > APIs. > I can see proto3 interfaces only for api, resource service and secret service. And I can see all the agents are mapped to switches on core [2][3] which talk to java dependencies. That's why I was thinking that way. Can you reference me where the agent interfaces are. > >> 3) The agent being independent is required for the agent to be converted >> to golang too. >> >> > Yes agent has to be independent from the code level with other components > of the framework but it should have API level dependencies to communicate > with the rest of the system. These are implemented on top of protobuf for > Resource and Secret services which can be easily built for golang and > consul level messaging dependencies with the controller which can be > fulfilled through a golang consul client or just following consul HTTP API. > Yes, resource service and secret service can be access via API. My question is how can I make the agent written in golang connect to the core [2][3] unless it's mapped via a REST api. > > 4) I was also thinking whether the combination of transports available per >> agent also should be independently installable. So, that each agent could >> decide which transports are required. Plus they could introduce new >> transport plugins. (A kind of an enhancement) >> > > +1 and this is a good feature to have in agents. Agent have a feature [1] > to inform controller about supported protocols but this not still validated > from the controller when submitting a transfer job. We need to implement > this feature > >> 5) I'm just doubt about the "LOCAL" transport. how does it work with >> multiple agents. According to the architecture where does the local storage >> exist? Is it inside the agent? >> > > Local resource is attached to the Agent. If and Agent receives a local > resource transfer request, it looks at the local file system of the machine > where it was installed. > Could we have such scenarios where the source or destination could be a storage attached to an agent server storage? Just curious to know. *References* [1] https://github.com/apache/airavata-mft/blob/master/agent/src/main/java/org/apache/airavata/mft/agent/MFTAgent.java#L245 [2] https://github.com/apache/airavata-mft/blob/master/core/src/main/java/org/apache/airavata/mft/core/ConnectorResolver.java#L29 [3] https://github.com/apache/airavata-mft/blob/master/core/src/main/java/org/apache/airavata/mft/core/MetadataCollectorResolver.java#L31 > > >> Regards, >> Dinuka >> >
