Re: [airavata-mft] Separating mft-agent module to a golang implementation
On Sun, Apr 19, 2020 at 9:22 PM DImuthu Upeksha wrote: > Hi Dinuka, > > I will reply inline > > On Fri, Apr 17, 2020 at 12:37 PM Dinuka Desilva > wrote: > >> Hi, >> >> I'm currently looking in to convert the existing java implementation of >> agent module to golang considering better concurrency, performance and make >> it more platform enabled. Before just starting up, I'm trying to >> discuss which kind of changes has to be done on which end. >> >> 1) I believe we are considering to accommodate multiple agents available >> per controller and each agent to support multiple transports. Which means, >> it's a collection of agents connected to a one controller. I think this is >> not done yet. >> >> There should be only one active controller for entire system. Controller > is the master and agents are the workers. This follows the master slave > architecture. Controller sends messages to agents when a transfer needs to > be performed. And this is already in place. > Got it. > > 2) Currently my understanding according to the implementation is that the >> agent module is kind of coupled with the api and controller. I feel that >> agent has to be an independent module. Or am I missing something? >> >> > Agent is not coupled with the controller. But it has a standard messaging > API built on top of Consul. If controller needs to get some work done by > the agent, it has to communicate in that particular messaging API. But > there is no code level dependency between the controller and agents. > However Agent has dependencies with Resource and Secret APIs to fetch > metadata of the resources. This is also not with the Service but with the > APIs. > I can see proto3 interfaces only for api, resource service and secret service. And I can see all the agents are mapped to switches on core [2][3] which talk to java dependencies. That's why I was thinking that way. Can you reference me where the agent interfaces are. > >> 3) The agent being independent is required for the agent to be converted >> to golang too. >> >> > Yes agent has to be independent from the code level with other components > of the framework but it should have API level dependencies to communicate > with the rest of the system. These are implemented on top of protobuf for > Resource and Secret services which can be easily built for golang and > consul level messaging dependencies with the controller which can be > fulfilled through a golang consul client or just following consul HTTP API. > Yes, resource service and secret service can be access via API. My question is how can I make the agent written in golang connect to the core [2][3] unless it's mapped via a REST api. > > 4) I was also thinking whether the combination of transports available per >> agent also should be independently installable. So, that each agent could >> decide which transports are required. Plus they could introduce new >> transport plugins. (A kind of an enhancement) >> > > +1 and this is a good feature to have in agents. Agent have a feature [1] > to inform controller about supported protocols but this not still validated > from the controller when submitting a transfer job. We need to implement > this feature > >> 5) I'm just doubt about the "LOCAL" transport. how does it work with >> multiple agents. According to the architecture where does the local storage >> exist? Is it inside the agent? >> > > Local resource is attached to the Agent. If and Agent receives a local > resource transfer request, it looks at the local file system of the machine > where it was installed. > Could we have such scenarios where the source or destination could be a storage attached to an agent server storage? Just curious to know. *References* [1] https://github.com/apache/airavata-mft/blob/master/agent/src/main/java/org/apache/airavata/mft/agent/MFTAgent.java#L245 [2] https://github.com/apache/airavata-mft/blob/master/core/src/main/java/org/apache/airavata/mft/core/ConnectorResolver.java#L29 [3] https://github.com/apache/airavata-mft/blob/master/core/src/main/java/org/apache/airavata/mft/core/MetadataCollectorResolver.java#L31 > > >> Regards, >> Dinuka >> >
Re: [airavata-mft] Separating mft-agent module to a golang implementation
Hi Dinuka, I will reply inline On Fri, Apr 17, 2020 at 12:37 PM Dinuka Desilva wrote: > Hi, > > I'm currently looking in to convert the existing java implementation of > agent module to golang considering better concurrency, performance and make > it more platform enabled. Before just starting up, I'm trying to > discuss which kind of changes has to be done on which end. > > 1) I believe we are considering to accommodate multiple agents available > per controller and each agent to support multiple transports. Which means, > it's a collection of agents connected to a one controller. I think this is > not done yet. > > There should be only one active controller for entire system. Controller is the master and agents are the workers. This follows the master slave architecture. Controller sends messages to agents when a transfer needs to be performed. And this is already in place. 2) Currently my understanding according to the implementation is that the > agent module is kind of coupled with the api and controller. I feel that > agent has to be an independent module. Or am I missing something? > > Agent is not coupled with the controller. But it has a standard messaging API built on top of Consul. If controller needs to get some work done by the agent, it has to communicate in that particular messaging API. But there is no code level dependency between the controller and agents. However Agent has dependencies with Resource and Secret APIs to fetch metadata of the resources. This is also not with the Service but with the APIs. > 3) The agent being independent is required for the agent to be converted > to golang too. > > Yes agent has to be independent from the code level with other components of the framework but it should have API level dependencies to communicate with the rest of the system. These are implemented on top of protobuf for Resource and Secret services which can be easily built for golang and consul level messaging dependencies with the controller which can be fulfilled through a golang consul client or just following consul HTTP API. 4) I was also thinking whether the combination of transports available per > agent also should be independently installable. So, that each agent could > decide which transports are required. Plus they could introduce new > transport plugins. (A kind of an enhancement) > +1 and this is a good feature to have in agents. Agent have a feature [1] to inform controller about supported protocols but this not still validated from the controller when submitting a transfer job. We need to implement this feature [1] https://github.com/apache/airavata-mft/blob/master/agent/src/main/java/org/apache/airavata/mft/agent/MFTAgent.java#L245 > > 5) I'm just doubt about the "LOCAL" transport. how does it work with > multiple agents. According to the architecture where does the local storage > exist? Is it inside the agent? > Local resource is attached to the Agent. If and Agent receives a local resource transfer request, it looks at the local file system of the machine where it was installed. > Regards, > Dinuka >
[airavata-mft] Separating mft-agent module to a golang implementation
Hi, I'm currently looking in to convert the existing java implementation of agent module to golang considering better concurrency, performance and make it more platform enabled. Before just starting up, I'm trying to discuss which kind of changes has to be done on which end. 1) I believe we are considering to accommodate multiple agents available per controller and each agent to support multiple transports. Which means, it's a collection of agents connected to a one controller. I think this is not done yet. 2) Currently my understanding according to the implementation is that the agent module is kind of coupled with the api and controller. I feel that agent has to be an independent module. Or am I missing something? 3) The agent being independent is required for the agent to be converted to golang too. 4) I was also thinking whether the combination of transports available per agent also should be independently installable. So, that each agent could decide which transports are required. Plus they could introduce new transport plugins. (A kind of an enhancement) 5) I'm just doubt about the "LOCAL" transport. how does it work with multiple agents. According to the architecture where does the local storage exist? Is it inside the agent? Regards, Dinuka