This is an automated email from the ASF dual-hosted git repository.
klesh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
The following commit(s) were added to refs/heads/main by this push:
new 0583a82 docs: update architecture page (#73)
0583a82 is described below
commit 0583a82b687d5339e48d2db449309b41c8da4c65
Author: Hezheng Yin <[email protected]>
AuthorDate: Thu Jun 9 06:52:32 2022 -0700
docs: update architecture page (#73)
---
docs/01-Overview/02-Architecture.md | 39 +++++++++++++++++++++++--------------
static/img/arch-component.svg | 2 +-
static/img/arch-dataflow.svg | 2 +-
3 files changed, 26 insertions(+), 17 deletions(-)
diff --git a/docs/01-Overview/02-Architecture.md
b/docs/01-Overview/02-Architecture.md
index 18d9b9a..8daa859 100755
--- a/docs/01-Overview/02-Architecture.md
+++ b/docs/01-Overview/02-Architecture.md
@@ -5,26 +5,35 @@ description: >
Understand the architecture of Apache DevLake.
---
-
-<p align="center"><img src="/img/arch-dataflow.svg" /></p>
-<p align="center">Architecture Diagram (data-flow perspective)</p>
+## Architecture Overview
<p align="center"><img src="/img/arch-component.svg" /></p>
-<p align="center">Architecture Diagram (component perspective)</p>
+<p align="center">DevLake Components</p>
+
+A DevLake installation typically consists of the following components:
+
+- Config UI: A handy user interface to create, trigger, and debug Blueprints.
A Blueprint specifies the where (data connection), what (data scope), how
(transformation rule), and when (sync frequency) of a data pipeline.
+- API Server: The main programmatic interface of DevLake.
+- Runner: The runner does all the heavy-lifting for executing tasks. In the
default DevLake installation, it runs within the API Server, but DevLake
provides a temporal-based runner (beta) for production environments.
+- Database: The database stores both DevLake's metadata and user data
collected by data pipelines. DevLake supports MySQL and PostgreSQL as of v0.11.
+- Plugins: Plugins enable DevLake to collect and analyze dev data from any
DevOps tools with an accessible API. DevLake community is actively adding
plugins for popular DevOps tools, but if your preferred tool is not covered
yet, feel free to open a GitHub issue to let us know or check out our doc on
how to build a new plugin by yourself.
+- Dashboards: Dashboards deliver data and insights to DevLake users. A
dashboard is simply a collection of SQL queries along with corresponding
visualization configurations. DevLake's official dashboard tool is Grafana and
pre-built dashboards are shipped in Grafana's JSON format. Users are welcome to
swap for their own choice of dashboard/BI tool if desired.
+
+## Dataflow
+
+<p align="center"><img src="/img/arch-dataflow.svg" /></p>
+<p align="center">DevLake Dataflow</p>
+A typical plugin's dataflow is illustrated below:
-## Stack (from low to high)
+1. The Raw layer stores the API responses from data sources (DevOps tools) in
JSON. This saves developers' time if the raw data is to be transformed
differently later on. Please note that communicating with data sources' APIs is
usually the most time-consuming step.
+2. The Tool layer extracts raw data from JSONs into a relational schema that's
easier to consume by analytical tasks. Each DevOps tool would have a schema
that's tailored to their data structure, hence the name, the Tool layer.
+3. The Domain layer attempts to build a layer of abstraction on top of the
Tool layer so that analytics logics can be re-used across different tools. For
example, GitHub's Pull Request (PR) and GitLab's Merge Request (MR) are similar
entities. They each have their own table name and schema in the Tool layer, but
they're consolidated into a single entity in the Domain layer, so that
developers only need to implement metrics like Cycle Time and Code Review
Rounds once against the domain la [...]
-1. config
-2. logger
-3. models
-4. plugins
-5. services
-6. api / cli
+## Principles
-## Rules
+1. Extensible: DevLake's plugin system allows users to integrate with any
DevOps tool. DevLake also provides a dbt plugin that enables users to define
their own data transformation and analysis workflows.
+2. Portable: DevLake has a modular design and provides multiple options for
each module. Users of different setups can freely choose the right
configuration for themselves.
+3. Robust: DevLake provides an SDK to help plugins efficiently and reliably
collect data from data sources while respecting their API rate limits and
constraints.
-1. Higher layers call lower layers, not the other way around
-2. Whenever a lower layer needs something from a higher layer, an interface
should be introduced for decoupling
-3. Components should be initialized in a low to high order during bootstrapping
<br/>
diff --git a/static/img/arch-component.svg b/static/img/arch-component.svg
index 5f9daa8..e060260 100644
--- a/static/img/arch-component.svg
+++ b/static/img/arch-component.svg
@@ -1,4 +1,4 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- Do not edit this file with editors other than diagrams.net -->
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd";>
-<svg xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink"; version="1.1" width="711px"
height="271px" viewBox="-0.5 -0.5 711 271" content="<mxfile
host="Electron" modified="2022-05-19T04:29:17.263Z"
agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
draw.io/18.0.1 Chrome/100.0.4896.143 Electron/18.2.0 Safari/537.36"
etag="Gl3qT091Rh7o-7RdCQlT" version="18.0.1"
type="device"><di [...]
\ No newline at end of file
+<svg xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink"; version="1.1" width="1182px"
height="662px" viewBox="-0.5 -0.5 1182 662" content="<mxfile
host="app.diagrams.net" modified="2022-06-08T21:46:25.152Z"
agent="5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/101.0.4951.64 Safari/537.36" version="19.0.3"
etag="6BozePPGZBuusqt-lJgN" type="google"><diagram
[...]
\ No newline at end of file
diff --git a/static/img/arch-dataflow.svg b/static/img/arch-dataflow.svg
index 8c0dff5..b7cf024 100644
--- a/static/img/arch-dataflow.svg
+++ b/static/img/arch-dataflow.svg
@@ -1,4 +1,4 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- Do not edit this file with editors other than diagrams.net -->
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd";>
-<svg xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink"; version="1.1" width="1678px"
height="918px" viewBox="-0.5 -0.5 1678 918" content="<mxfile
host="Electron" modified="2022-05-11T13:39:34.016Z"
agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
draw.io/17.4.2 Chrome/100.0.4896.60 Electron/18.0.1 Safari/537.36"
etag="ZEm8dapN1nsaMz5g2JYS" version="17.4.2"
type="device"><d [...]
\ No newline at end of file
+<svg xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink"; version="1.1" width="1197px"
height="982px" viewBox="-0.5 -0.5 1197 982" content="<mxfile
host="app.diagrams.net" modified="2022-06-08T21:43:37.556Z"
agent="5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/101.0.4951.64 Safari/537.36"
etag="0UrxqShNez2ZaexYe9pV" version="19.0.3"
type="google"><diagram [...]
\ No newline at end of file
