Hi all Adding helix-dev as this is mainly focused on deploying Helix Tasks in a containerized environment.
To provide a general overview, I'm implementing a task execution framework for Airavata using the task framework of Helix. I'm using Helix tasks as the constructs that synthesize a workflow of Airavata. Example for possible tasks are executing a scientific job on a remote resource, pushing input files to scientific jobs and fetching outputs once a job is completed. Once I came up with a working prototype [1], my next consideration was about the deployment aspect of these tasks. I chose to package these tasks as docker containers and deploy in a Kubernetes cluster. Helix Task components were packaged as *one task per participant and one participant per container* (pod). By doing that, I had to specify an instance group tag for each participant (container) in order to find correct participant when scheduling a task by Helix controller. Diagram: https://drive.google.com/file/d/1P1JGgmzx7yttygpxnZhuApbJMzMlafs5/view?usp=sharing This enables us to independently scale and update each task type without interrupting other tasks in the framework. If we want to scale a particular task type, we can command to Kubernetes API with a replication count and Kubernetes brings up the number of replications of the task to the specified level. Next problem is, what do we provide as the names of the participants which are deployed as containers. I used the host name of the pod which is unique along the Kubernetes cluster. This makes sure that two participants with the same name do not exist. You can find Kubernetes scripts that perform above mentioned deployment from here [2] and it will be really helpful if @heix-dev can provide feedback or the best practices upon the above approach of containerizing Helix tasks and participants. [1] https://github.com/DImuthuUpe/airavata-sandbox/tree/master/airavata- kubernetes/modules/microservices/tasks [2] https://github.com/DImuthuUpe/airavata-sandbox/tree/master/airavata- kubernetes/scripts/k8s Thanks
