This is an automated email from the ASF dual-hosted git repository.
yasith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata.git
The following commit(s) were added to refs/heads/master by this push:
new be607c4d97 update README with detailed description of internals
be607c4d97 is described below
commit be607c4d9773ac34760cd1097262f8ba4c93bddf
Author: Yasith Jayawardana <[email protected]>
AuthorDate: Fri Jun 27 02:36:56 2025 -0500
update README with detailed description of internals
---
README.md | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 65 insertions(+), 9 deletions(-)
diff --git a/README.md b/README.md
index 5c328f51d5..fc6b6fabba 100644
--- a/README.md
+++ b/README.md
@@ -20,6 +20,8 @@ Apache Airavata is a software framework for executing and
managing computational
> Learn more at [airavata.apache.org](https://airavata.apache.org)
+
+
## 🧱 The Airavata Ecosystem
Apache Airavata is composed of modular components spanning core services, data
management, user interfaces, and developer tooling.
@@ -103,18 +105,72 @@ ls -la modules/distribution/target/
```
-### Core Service Classes
+### Core Airavata Services
+
+
+
+
+#### API Server
+> Class Name: `org.apache.airavata.server.ServerMain`
+
+The API Server is the entrypoint to Airavata, bootstrapping 7 core services
that run in parallel. Each service implements the
`org.apache.airavata.common.utils.IServer` interface.
+
+- API - `org.apache.airavata.api.server.AiravataAPIServer` - public-facing API
that is consumed by Airavata SDKs and dashboards. It bridges external clients
to internal Airavata services, and is served over Thrift.
+- DB Event Manager -
`org.apache.airavata.db.event.manager.DBEventManagerRunner` - Monitors task
execution events (launch, transitions, completion/failure) and syncs them to
the Airavata DB via pub/sub hooks.
+- Registry - `org.apache.airavata.registry.api.service.RegistryAPIServer` -
Manages metadata and definitions for executable tasks and applications.
+- Credential Store -
`org.apache.airavata.credential.store.server.CredentialStoreServer` - Manages
secure storage and retrieval of credentials for accessing registered compute
resources.
+- Sharing Registry -
`org.apache.airavata.sharing.registry.server.SharingRegistryServer` - Handles
sharing and permissioning of Airavata resources between users and groups.
+- Orchestrator - `org.apache.airavata.orchestrator.server.OrchestratorServer`
- Constructs workflow DAGs, assigns unique IDs to tasks, and hands them off to
the workflow manager.
+- Profile - `org.apache.airavata.service.profile.server.ProfileServiceServer`
- Manages users, tenants, compute resources, and group profiles.
+
+#### Pre Workflow Manager
+> Class Name: `org.apache.airavata.helix.impl.workflow.PreWorkflowManager`
+
+The pre-workflow manager listens on the internal MQ (KafkaConsumer) to inbound
tasks at **pre-execution** phase. When a task DAG is received, it handles the
environment setup and data staging phases of the DAG in a robust manner, which
includes fault-handling. All these happen BEFORE the task DAG is submitted to
the controller, and subsequently to the participant.
+
+#### Controller
+> Class Name: `org.apache.airavata.helix.impl.controller.HelixController`
+
+The Controller manages the step-by-step transition of task state on
*helix-side*. It uses Apache Helix to track step start, completion, and failure
paths, ensuring the next step starts upon successful completion or retrying the
current step on failure.
+
+#### Participant
+> Class Name: `org.apache.airavata.helix.impl.participant.GlobalParticipant`
+
+The participant synchronizes the *helix-side* state transition of a task with
its concrete execution at *airavata-side*. The currently registered steps are:
`EnvSetupTask`, `InputDataStagingTask`, `OutputDataStagingTask`,
`JobVerificationTask`, `CompletingTask`, `ForkJobSubmissionTask`,
`DefaultJobSubmissionTask`, `LocalJobSubmissionTask`, `ArchiveTask`,
`WorkflowCancellationTask`, `RemoteJobCancellationTask`,
`CancelCompletingTask`, `DataParsingTask`, `ParsingTriggeringTask`, and
`MockTask`.
+
+#### Post Workflow Manager
+> Class Name: `org.apache.airavata.helix.impl.workflow.PostWorkflowManager`
+
+The post-workflow listens on the internal MQ (KafkaConsumer) to inbound tasks
at **post-execution** phase. Once a task is received, it handles the cleanup
and output fetching phases of the task DAG in a robust manner, which includes
fault-handling. Once the main task completes executing, this is announced to
the realtime monitor, upon which the post-workflow phase is triggered. Once
triggered, it submits this state change to the controller.
+
+#### Parser Workflow Manager (Deprecated)
+> Class Name: `org.apache.airavata.helix.impl.workflow.ParserWorkflowManager`
+
+The parser-workflow listens on the internal MQ (KafkaConsumer) to inbound
tasks at **post-completion** phase., which includes transforming generated
outputs into different formats. This component is not actively used in airavata.
+
+
+
+#### Email Monitor
+> Class Name: `org.apache.airavata.monitor.email.EmailBasedMonitor`
+
+The email monitor periodically checks an email inbox for job status updates
sent via email. If it reads a new email with a job status update, it relays
that state-change to the internal MQ (KafkaProducer).
+
+#### Realtime Monitor
+> Class Name: `org.apache.airavata.monitor.realtime.RealtimeMonitor`
+
+The realtime monitor listens to incoming state-change messages on the internal
MQ (KafkaConsumer), and relays that state-change to the internal MQ
(KafkaProducer). When a task is completed at the compute resource, the realtime
monitor is notified of this.
+
+#### Agent Service
+> Class Name:
`org.apache.airavata.agent.connection.service.AgentServiceApplication`
-* API Server - `org.apache.airavata.server.ServerMain`
-* Controller - `org.apache.airavata.helix.impl.controller.HelixController`
-* Participant - `org.apache.airavata.helix.impl.participant.GlobalParticipant`
+The agent service is the backend for launching interactive jobs using Airavata.
+It provide constructs to launch a custom "Agent" on a compute resource, that
connects back to the Agent Service through a bi-directional gRPC channel.
+The Airavata Python SDK primarily utilizes the Agent Service (gRPC) and the
Airavata API (Thrift) to submit and execute interactive jobs, spawn
subprocesses, and create network tunnels to subprocesses, even if they are
behind NAT.
-* Pre WM - `org.apache.airavata.helix.impl.workflow.PreWorkflowManager`
-* Parser WM - `org.apache.airavata.helix.impl.workflow.ParserWorkflowManager`
-* Post WM - `org.apache.airavata.helix.impl.workflow.PostWorkflowManager`
+#### Research Service
+> Class Name: `org.apache.airavata.research.service.ResearchServiceApplication`
-* Email Monitor - `org.apache.airavata.monitor.email.EmailBasedMonitor`
-* Realtime Monitor - `org.apache.airavata.monitor.realtime.RealtimeMonitor`
+The research service is the backend for the Airavata research catalog. It
provides the API to add, list, modify, and publish notebooks, repositories,
datasets, and computational models in cybershuttle, and launch interactive
remote sessions to utilize them in a research setting.
### 🐳 Docker Development (Experimental)
