This is an automated email from the ASF dual-hosted git repository.
klesh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
The following commit(s) were added to refs/heads/main by this push:
new cf73c05 doc: add repo readmes to website (#23)
cf73c05 is described below
commit cf73c0523a30f2e69c2c3c95e6d57783dab8e82b
Author: Louis.z <[email protected]>
AuthorDate: Thu May 19 00:01:53 2022 +0800
doc: add repo readmes to website (#23)
* doc: add repo readmes to website
* fix: move the image into the right folder
Co-authored-by: Startrekzky <[email protected]>
---
docs/02-QuickStart/01-LocalSetup.md | 98 +++------
docs/02-QuickStart/02-KubernetesSetup.md | 35 ++-
docs/02-QuickStart/03-DeveloperSetup.md | 59 ++---
docs/03-Features.md | 20 --
docs/03-Plugins/_category_.json | 4 +
docs/03-Plugins/dbt.md | 65 ++++++
docs/03-Plugins/feishu.md | 65 ++++++
docs/03-Plugins/gitextractor.md | 64 ++++++
docs/03-Plugins/github.md | 102 +++++++++
docs/03-Plugins/gitlab.md | 92 ++++++++
docs/03-Plugins/jenkins.md | 64 ++++++
docs/03-Plugins/jira.md | 242 +++++++++++++++++++++
docs/03-Plugins/refdiff.md | 115 ++++++++++
docs/04-UserManuals/GRAFANA.md | 119 ++++++++++
docs/04-UserManuals/MIGRATIONS.md | 36 +++
docs/04-UserManuals/NOTIFICATION.md | 33 +++
docs/04-UserManuals/_category_.json | 4 +
.../create-pipeline-in-advanced-mode.md | 88 ++++++++
docs/04-UserManuals/github-user-guide-v0.10.0.md | 119 ++++++++++
docs/04-UserManuals/recurring-pipeline.md | 29 +++
.../01-DevLakeDomainLayerSchema.md | 0
.../02-DataSupport.md | 0
.../_category_.json | 2 +-
...ineeringMetrics.md => 06-EngineeringMetrics.md} | 2 +-
docs/{06-Glossary.md => 07-Glossary.md} | 26 +--
docs/{07-LiveDemo.md => 08-LiveDemo.md} | 2 +-
static/img/github-demo.png | Bin 0 -> 45040 bytes
static/img/jenkins-demo.png | Bin 0 -> 128996 bytes
static/img/jira-demo.png | Bin 0 -> 622316 bytes
29 files changed, 1347 insertions(+), 138 deletions(-)
diff --git a/docs/02-QuickStart/01-LocalSetup.md
b/docs/02-QuickStart/01-LocalSetup.md
index 98ed3bb..eb47e59 100644
--- a/docs/02-QuickStart/01-LocalSetup.md
+++ b/docs/02-QuickStart/01-LocalSetup.md
@@ -1,81 +1,43 @@
---
-title: "Local Setup"
-linkTitle: "Local Setup"
-tags: []
-categories: []
-weight: 1
+title: "Deploy Locally"
description: >
The steps to install DevLake locally.
---
-- If you only plan to run the product locally, this is the **ONLY** section
you should need.
-- Commands written `like this` are to be run in your terminal.
-
-#### Required Packages to Install<a id="user-setup-requirements"></a>
-
-- [Docker](https://docs.docker.com/get-docker)
-- [docker-compose](https://docs.docker.com/compose/install/)
+#### Prerequisites
-NOTE: After installing docker, you may need to run the docker application and
restart your terminal
+- [Docker v19.03.10+](https://docs.docker.com/get-docker)
+- [docker-compose v2.2.3+](https://docs.docker.com/compose/install/)
-#### Commands to run in your terminal<a id="user-setup-commands"></a>
+#### Launch DevLake
-**IMPORTANT: DevLake doesn't support Database Schema Migration yet, upgrading
an existing instance is likely to break, we recommend that you deploy a new
instance instead.**
+- Commands written `like this` are to be run in your terminal.
-1. Download `docker-compose.yml` and `env.example` from [latest release
page](https://github.com/merico-dev/lake/releases/latest) into a folder.
+1. Download `docker-compose.yml` and `env.example` from [latest release
page](https://github.com/apache/incubator-devlake/releases/latest) into a
folder.
2. Rename `env.example` to `.env`. For Mac/Linux users, please run `mv
env.example .env` in the terminal.
-3. Start Docker on your machine, then run `docker-compose up -d` to start the
services.
-4. Visit `localhost:4000` to set up configuration files.
- >- Navigate to desired plugins on the Integrations page
- >- Please reference the following for more details on how to configure each
one:<br/>
- - **jira**
- - **gitlab**
- - **jenkins**
- - **github**
- >- Submit the form to update the values by clicking on the **Save
Connection** button on each form page
- >- `devlake` takes a while to fully boot up. if `config-ui` complaining
about api being unreachable, please wait a few seconds and try refreshing the
page.
-
-
-5. Visit `localhost:4000/pipelines/create` to RUN a Pipeline and trigger data
collection.
-
-
- Pipelines Runs can be initiated by the new "Create Run" Interface. Simply
enable the **Data Source Providers** you wish to run collection for, and
specify the data you want to collect, for instance, **Project ID** for Gitlab
and **Repository Name** for GitHub.
-
- Once a valid pipeline configuration has been created, press **Create Run**
to start/run the pipeline.
- After the pipeline starts, you will be automatically redirected to the
**Pipeline Activity** screen to monitor collection activity.
-
- **Pipelines** is accessible from the main menu of the config-ui for easy
access.
-
- - Manage All Pipelines: `http://localhost:4000/pipelines`
- - Create Pipeline RUN: `http://localhost:4000/pipelines/create`
- - Track Pipeline Activity:
`http://localhost:4000/pipelines/activity/[RUN_ID]`
-
- For advanced use cases and complex pipelines, please use the Raw JSON API
to manually initiate a run using **cURL** or graphical API tool such as
**Postman**. `POST` the following request to the DevLake API Endpoint.
-
- ```json
- [
- [
- {
- "plugin": "github",
- "options": {
- "repo": "lake",
- "owner": "merico-dev"
- }
- }
- ]
- ]
- ```
-
- Please refer to this wiki [How to trigger data
collection](https://github.com/merico-dev/lake/wiki/How-to-use-the-triggers-page).
-
-6. Click *View Dashboards* button in the top left when done, or visit
`localhost:3002` (username: `admin`, password: `admin`).
-
- We use <a href="https://grafana.com/"; target="_blank">Grafana</a> as a
visualization tool to build charts for the <a
href="https://github.com/merico-dev/lake/wiki/DataModel.Domain-layer-schema";>data
stored in our database</a>. Using SQL queries, we can add panels to build,
save, and edit customized dashboards.
-
- All the details on provisioning and customizing a dashboard can be found in
the **Grafana Doc**
-
-#### Setup cron job
+3. Run `docker-compose up -d` to launch DevLake.
+
+#### Configure data connections and collect data
+
+1. Visit `config-ui` at `http://localhost:4000` in your browser to configure
data connections.
+ - Navigate to desired plugins on the Integrations page
+ - Please reference the following for more details on how to configure each
one:<br/>
+ - [Jira](../03-Plugins/jira.md)
+ - [GitHub](../03-Plugins/github.md): For users who'd like to collect
GitHub data, we recommend reading our [GitHub data collection
guide](../04-UserManuals/github-user-guide-v0.10.0.md) which covers the
following steps in detail.
+ - [GitLab](../03-Plugins/gitlab.md)
+ - [Jenkins](../03-Plugins/jenkins.md)
+ - Submit the form to update the values by clicking on the **Save
Connection** button on each form page
+ - `devlake` takes a while to fully boot up. if `config-ui` complaining
about api being unreachable, please wait a few seconds and try refreshing the
page.
+2. Create pipelines to trigger data collection in `config-ui`
+3. Click *View Dashboards* button in the top left when done, or visit
`localhost:3002` (username: `admin`, password: `admin`).
+ - We use [Grafana](https://grafana.com/) as a visualization tool to build
charts for the [data](../05-DataModels/02-DataSupport.md) stored in our
database.
+ - Using SQL queries, we can add panels to build, save, and edit customized
dashboards.
+ - All the details on provisioning and customizing a dashboard can be found
in the [Grafana Doc](../04-UserManuals/GRAFANA.md).
+4. To synchronize data periodically, users can set up recurring pipelines with
DevLake's [pipeline blueprint](../04-UserManuals/recurring-pipeline.md) for
details.
+
+#### Upgrade to a newer version
+
+Support for database schema migration was introduced to DevLake in v0.10.0.
From v0.10.0 onwards, users can upgrade their instance smoothly to a newer
version. However, versions prior to v0.10.0 do not support upgrading to a newer
version with a different database schema. We recommend users deploying a new
instance if needed.
-To synchronize data periodically, we provide **lake-cli** for easily sending
data collection requests along with **a cron job** to periodically trigger the
cli tool.
<br/><br/><br/>
diff --git a/docs/02-QuickStart/02-KubernetesSetup.md
b/docs/02-QuickStart/02-KubernetesSetup.md
index 4327e31..4e65d69 100644
--- a/docs/02-QuickStart/02-KubernetesSetup.md
+++ b/docs/02-QuickStart/02-KubernetesSetup.md
@@ -1,9 +1,32 @@
---
-title: "Kubernetes Setup"
-linkTitle: "Kubernetes Setup"
-tags: []
-categories: []
-weight: 2
+title: "Deploy to Kubernates"
description: >
- The steps to install DevLake in Kubernetes.
+ The steps to install Apache DevLake in Kubernetes.
---
+
+
+We provide a sample
[k8s-deploy.yaml](https://github.com/apache/incubator-devlake/blob/main/k8s-deploy.yaml)
for users interested in deploying Apache DevLake on a k8s cluster.
+
+[k8s-deploy.yaml](https://github.com/apache/incubator-devlake/blob/main/k8s-deploy.yaml)
will create a namespace `devlake` on your k8s cluster, and use `nodePort
30004` for `config-ui`, `nodePort 30002` for `grafana` dashboards. If you
would like to use certain version of Apache DevLake, please update the image
tag of `grafana`, `devlake` and `config-ui` services to specify versions like
`v0.10.1`.
+
+Here's the step-by-step guide:
+
+1. Download
[k8s-deploy.yaml](https://github.com/apache/incubator-devlake/blob/main/k8s-deploy.yaml)
to local machine
+2. Some key points:
+ - `config-ui` deployment:
+ * `GRAFANA_ENDPOINT`: FQDN of grafana service which can be reached from
user's browser
+ * `DEVLAKE_ENDPOINT`: FQDN of devlake service which can be reached within
k8s cluster, normally you don't need to change it unless namespace was changed
+ * `ADMIN_USER`/`ADMIN_PASS`: Not required, but highly recommended
+ - `devlake-config` config map:
+ * `MYSQL_USER`: shared between `mysql` and `grafana` service
+ * `MYSQL_PASSWORD`: shared between `mysql` and `grafana` service
+ * `MYSQL_DATABASE`: shared between `mysql` and `grafana` service
+ * `MYSQL_ROOT_PASSWORD`: set root password for `mysql` service
+ - `devlake` deployment:
+ * `DB_URL`: update this value if `MYSQL_USER`, `MYSQL_PASSWORD` or
`MYSQL_DATABASE` were changed
+3. The `devlake` deployment store its configuration in `/app/.env`. In our
sample yaml, we use `hostPath` volume, so please make sure directory
`/var/lib/devlake` exists on your k8s workers, or employ other techniques to
persist `/app/.env` file. Please do NOT mount the entire `/app` directory,
because plugins are located in `/app/bin` folder.
+4. Finally, execute the following command, Apache DevLake should be up and
running:
+ ```sh
+ kubectl apply -f k8s-deploy.yaml
+ ```
+<br/><br/><br/>
diff --git a/docs/02-QuickStart/03-DeveloperSetup.md
b/docs/02-QuickStart/03-DeveloperSetup.md
index b1696d0..61145b5 100644
--- a/docs/02-QuickStart/03-DeveloperSetup.md
+++ b/docs/02-QuickStart/03-DeveloperSetup.md
@@ -1,23 +1,18 @@
---
title: "Developer Setup"
-linkTitle: "Developer Setup"
-tags: []
-categories: []
-weight: 3
description: >
The steps to install DevLake in develper mode.
---
-
#### Requirements
-- <a href="https://docs.docker.com/get-docker"; target="_blank">Docker</a>
+- <a href="https://docs.docker.com/get-docker"; target="_blank">Docker
v19.03.10+</a>
- <a href="https://golang.org/doc/install"; target="_blank">Golang v1.17+</a>
- Make
- Mac (Already installed)
- Windows: [Download](http://gnuwin32.sourceforge.net/packages/make.htm)
- - Ubuntu: `sudo apt-get install build-essential`
+ - Ubuntu: `sudo apt-get install build-essential libssl-dev`
#### How to setup dev environment
1. Navigate to where you would like to install this project and clone the
repository:
@@ -29,7 +24,7 @@ description: >
2. Install dependencies for plugins:
- - **RefDiff**
+ - [RefDiff](../03-Plugins/refdiff.md#development)
3. Install Go packages
@@ -47,7 +42,7 @@ description: >
* `DB_URL`: Replace `mysql:3306` with `127.0.0.1:3306`
-5. Start the MySQL and Grafana containers:
+6. Start the MySQL and Grafana containers:
> Make sure the Docker daemon is running before this step.
@@ -55,7 +50,7 @@ description: >
docker-compose up -d mysql grafana
```
-6. Run lake and config UI in dev mode in two seperate terminals:
+7. Run lake and config UI in dev mode in two seperate terminals:
```sh
# run lake
@@ -64,21 +59,28 @@ description: >
make configure-dev
```
-7. Visit config UI at `localhost:4000` to configure data sources.
- >- Navigate to desired plugins pages on the Integrations page
- >- You will need to enter the required information for the plugins you
intend to use.
- >- Please reference the following for more details on how to configure each
one:
- **jira**
- **gitlab**
- **jenkins**
- **github**
+ Q: I got an error saying: `libgit2.so.1.3: cannot open share object file:
No such file or directory`
+
+ A: Make sure your program find `libgit2.so.1.3`. `LD_LIBRARY_PATH` can be
assigned like this if your `libgit2.so.1.3` is located at `/usr/local/lib`:
+
+ ```sh
+ export LD_LIBRARY_PATH=/usr/local/lib
+ ```
- >- Submit the form to update the values by clicking on the **Save
Connection** button on each form page
+8. Visit config UI at `localhost:4000` to configure data connections.
+ - Navigate to desired plugins pages on the Integrations page
+ - You will need to enter the required information for the plugins you
intend to use.
+ - Please reference the following for more details on how to configure each
one:
+ - [Jira](../03-Plugins/jira.md)
+ - [GitLab](../03-Plugins/gitlab.md)
+ - [Jenkins](../03-Plugins/jenkins.md)
+ - [GitHub](../03-Plugins/github.md): For users who'd like to collect
GitHub data, we recommend reading our [GitHub data collection
guide](../04-UserManuals/github-user-guide-v0.10.0.md) which covers the
following steps in detail.
+ - Submit the form to update the values by clicking on the **Save
Connection** button on each form page
-8. Visit `localhost:4000/pipelines/create` to RUN a Pipeline and trigger data
collection.
+9. Visit `localhost:4000/pipelines/create` to RUN a Pipeline and trigger data
collection.
- Pipelines Runs can be initiated by the new "Create Run" Interface. Simply
enable the **Data Source Providers** you wish to run collection for, and
specify the data you want to collect, for instance, **Project ID** for Gitlab
and **Repository Name** for GitHub.
+ Pipelines Runs can be initiated by the new "Create Run" Interface. Simply
enable the **Data Connection Providers** you wish to run collection for, and
specify the data you want to collect, for instance, **Project ID** for Gitlab
and **Repository Name** for GitHub.
Once a valid pipeline configuration has been created, press **Create Run**
to start/run the pipeline.
After the pipeline starts, you will be automatically redirected to the
**Pipeline Activity** screen to monitor collection activity.
@@ -105,19 +107,20 @@ description: >
]
```
- Please refer to this wiki [How to trigger data
collection](https://github.com/merico-dev/lake/wiki/How-to-use-the-triggers-page).
+ Please refer to [Pipeline Advanced
Mode](../04-UserManuals/create-pipeline-in-advanced-mode.md) for in-depth
explanation.
-9. Click *View Dashboards* button in the top left when done, or visit
`localhost:3002` (username: `admin`, password: `admin`).
+10. Click *View Dashboards* button in the top left when done, or visit
`localhost:3002` (username: `admin`, password: `admin`).
We use <a href="https://grafana.com/"; target="_blank">Grafana</a> as a
visualization tool to build charts for the <a
href="https://github.com/merico-dev/lake/wiki/DataModel.Domain-layer-schema";>data
stored in our database</a>. Using SQL queries, we can add panels to build,
save, and edit customized dashboards.
- All the details on provisioning and customizing a dashboard can be found in
the **Grafana Doc**
-
-
-10. (Optional) To run the tests:
+ All the details on provisioning and customizing a dashboard can be found in
the [Grafana Doc](../04-UserManuals/GRAFANA.md).
+11. (Optional) To run the tests:
+ ```sh
make test
+ ```
-<br/><br/><br/><br/>
+12. For DB migrations, please refer to [Migration
Doc](../04-UserManuals/MIGRATIONS.md).
+<br/><br/><br/>
diff --git a/docs/03-Features.md b/docs/03-Features.md
deleted file mode 100644
index 75f2a16..0000000
--- a/docs/03-Features.md
+++ /dev/null
@@ -1,20 +0,0 @@
----
-sidebar_position: 03
-title: "Features"
-linkTitle: "Features"
-tags: []
-categories: []
-weight: 30000
-description: >
- Features of the latest version of DevLake.
----
-
-
-1. Collect data from [mainstream DevOps
tools](https://github.com/merico-dev/lake#project-roadmap), including Jira
(Cloud), Jira Server v8+, Git, GitLab, GitHub, Jenkins etc., supported by
[plugins](https://github.com/merico-dev/lake/blob/main/ARCHITECTURE.md).
-2. Receive data through Push API.
-3. Standardize DevOps data based on [domain layer
schema](https://github.com/merico-dev/lake/wiki/DataModel.Domain-layer-schema).
Support 20+ [built-in engineering
metrics](https://github.com/merico-dev/lake/wiki/Metric-Cheatsheet) to observe
productivity, quality and delivery capability.
-4. Connect "commit" entity with "issue" entity, and generate composite metrics
such as `Bugs Count per 1k Lines of Code`.
-5. Identify new commits based on
[RefDiff](https://github.com/merico-dev/lake/tree/main/plugins/refdiff#refdiff)
plugin, and analyze productivity and quality of each version.
-6. Flexible dashboards to support data visualization and queries, based on
Grafana.
-
-<br/><br/><br/>
diff --git a/docs/03-Plugins/_category_.json b/docs/03-Plugins/_category_.json
new file mode 100644
index 0000000..36011ea
--- /dev/null
+++ b/docs/03-Plugins/_category_.json
@@ -0,0 +1,4 @@
+{
+ "label": "Plugins",
+ "position": 3
+}
diff --git a/docs/03-Plugins/dbt.md b/docs/03-Plugins/dbt.md
new file mode 100644
index 0000000..ebfd01a
--- /dev/null
+++ b/docs/03-Plugins/dbt.md
@@ -0,0 +1,65 @@
+---
+title: "DBT"
+description: >
+ DBT Plugin
+---
+
+
+## Summary
+
+dbt (data build tool) enables analytics engineers to transform data in their
warehouses by simply writing select statements. dbt handles turning these
select statements into tables and views.
+dbt does the T in ELT (Extract, Load, Transform) processes – it doesn’t
extract or load data, but it’s extremely good at transforming data that’s
already loaded into your warehouse.
+
+## User setup<a id="user-setup"></a>
+- If you plan to use this product, you need to install some environments first.
+
+#### Required Packages to Install<a id="user-setup-requirements"></a>
+- [python3.7+](https://www.python.org/downloads/)
+- [dbt-mysql](https://pypi.org/project/dbt-mysql/#configuring-your-profile)
+
+#### Commands to run or create in your terminal and the dbt project<a
id="user-setup-commands"></a>
+1. pip install dbt-mysql
+2. dbt init demoapp (demoapp is project name)
+3. create your SQL transformations and data models
+
+## Convert Data By DBT
+
+please use the Raw JSON API to manually initiate a run using **cURL** or
graphical API tool such as **Postman**. `POST` the following request to the
DevLake API Endpoint.
+
+```json
+[
+ [
+ {
+ "plugin": "dbt",
+ "options": {
+ "projectPath": "/Users/abeizn/demoapp",
+ "projectName": "demoapp",
+ "projectTarget": "dev",
+ "selectedModels": ["my_first_dbt_model","my_second_dbt_model"],
+ "projectVars": {
+ "demokey1": "demovalue1",
+ "demokey2": "demovalue2"
+ }
+ }
+ }
+ ]
+]
+```
+
+- `projectPath`: the absolute path of the dbt project. (required)
+- `projectName`: the name of the dbt project. (required)
+- `projectTarget`: this is the default target your dbt project will use.
(optional)
+- `selectedModels`: a model is a select statement. Models are defined in .sql
files, and typically in your models directory. (required)
+And selectedModels accepts one or more arguments. Each argument can be one of:
+1. a package name #runs all models in your project, example: example
+2. a model name # runs a specific model, example: my_fisrt_dbt_model
+3. a fully-qualified path to a directory of models.
+
+- `vars`: dbt provides a mechanism variables to provide data to models for
compilation. (optional)
+example: select * from events where event_type = '{{ var("event_type") }}'
this sql in your model, you need set parameters "vars": "{event_type:
real_value}"
+
+### Resources:
+- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
+- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked
questions and answers
+
+<br/><br/><br/>
diff --git a/docs/03-Plugins/feishu.md b/docs/03-Plugins/feishu.md
new file mode 100644
index 0000000..d05e930
--- /dev/null
+++ b/docs/03-Plugins/feishu.md
@@ -0,0 +1,65 @@
+---
+title: "Feishu"
+description: >
+ Feishu Plugin
+---
+
+# Feishu
+
+## Summary
+
+This plugin collects Feishu data through [Feishu
Openapi](https://open.feishu.cn/document/home/user-identity-introduction/introduction).
+
+## Configuration
+
+In order to fully use this plugin, you will need to get app_id and app_secret
from feishu administrator(For help on App info, please see [official Feishu
Docs](https://open.feishu.cn/document/ukTMukTMukTM/ukDNz4SO0MjL5QzM/auth-v3/auth/tenant_access_token_internal)),
+then set these two configurations via Dev Lake's `.env`.
+
+### By `.env`
+
+The connection aspect of the configuration screen requires the following key
fields to connect to the Feishu API. As Feishu is a single-source data provider
at the moment, the connection name is read-only as there is only one instance
to manage. As we continue our development roadmap we may enable multi-source
connections for Feishu in the future.
+
+FEISHU_APPID=app_id
+
+FEISHU_APPSCRECT=app_secret
+
+
+## Collect Data From Feishu
+
+In order to collect data, you have to compose a JSON looks like following one,
and send it by selecting `Advanced Mode` on `Create Pipeline Run` page:
+numOfDaysToCollect： The number of days you want to collect
+rateLimitPerSecond: The number of requests to send(Maximum is 8)
+
+```json
+[
+ [
+ {
+ "plugin": "feishu",
+ "options": {
+ "numOfDaysToCollect" : 80,
+ "rateLimitPerSecond" : 5
+ }
+ }
+ ]
+]
+```
+
+You can also trigger data collection by making a POST request to `/pipelines`.
+```
+curl --location --request POST 'localhost:8080/pipelines' \
+--header 'Content-Type: application/json' \
+--data-raw '
+ {
+ "name": "feishu 20211126",
+ "tasks": [[{
+ "plugin": "feishu",
+ "options": {
+ "numOfDaysToCollect" : 80,
+ "rateLimitPerSecond" : 5
+ }
+ }]]
+ }
+'
+```
+
+<br/><br/><br/>
diff --git a/docs/03-Plugins/gitextractor.md b/docs/03-Plugins/gitextractor.md
new file mode 100644
index 0000000..2057b06
--- /dev/null
+++ b/docs/03-Plugins/gitextractor.md
@@ -0,0 +1,64 @@
+---
+title: "GitExtractor"
+description: >
+ GitExtractor Plugin
+---
+
+# Git Repo Extractor
+
+## Summary
+This plugin extract commits and references from a remote or local git
repository. It then saves the data into the database or csv files.
+
+## Steps to make this plugin work
+
+1. Use the Git repo extractor to retrieve commit-and-branch-related data from
your repo
+2. Use the GitHub plugin to retrieve Github-issue-and-pr-related data from
your repo. NOTE: you can run only one the issue collection stage as described
in the Github Plugin README.
+3. Use the [RefDiff](./refdiff.md#development) plugin to calculate version
diff, which will be stored in `refs_commits_diffs` table.
+
+## Sample Request
+
+```
+curl --location --request POST 'localhost:8080/pipelines' \
+--header 'Content-Type: application/json' \
+--data-raw '
+{
+ "name": "git repo extractor",
+ "tasks": [
+ [
+ {
+ "Plugin": "gitextractor",
+ "Options": {
+ "url": "https://github.com/merico-dev/lake.git";,
+ "repoId": "github:GithubRepo:384111310"
+ }
+ }
+ ]
+ ]
+}
+'
+```
+- `url`: the location of the git repository. It should start with
`http`/`https` for remote git repository or `/` for a local one.
+- `repoId`: column `id` of `repos`.
+- `proxy`: optional, http proxy, e.g. `http://your-proxy-server.com:1080`.
+- `user`: optional, for cloning private repository using HTTP/HTTPS
+- `password`: optional, for cloning private repository using HTTP/HTTPS
+- `privateKey`: optional, for SSH cloning, base64 encoded `PEM` file
+- `passphrase`: optional, passphrase for the private key
+
+
+## Standalone Mode
+
+You call also run this plugin in a standalone mode without any DevLake service
running using the following command:
+
+```
+go run plugins/gitextractor/main.go -url
https://github.com/merico-dev/lake.git -id github:GithubRepo:384111310 -db
"merico:merico@tcp(127.0.0.1:3306)/lake?charset=utf8mb4&parseTime=True"
+```
+
+For more options (e.g., saving to a csv file instead of a db), please read
`plugins/gitextractor/main.go`.
+
+## Development
+
+This plugin depends on `libgit2`, you need to install version 1.3.0 in order
to run and debug this plugin on your local
+machine. [Click here](./refdiff.md#development) for a brief guide.
+
+<br/><br/><br/>
diff --git a/docs/03-Plugins/github.md b/docs/03-Plugins/github.md
new file mode 100644
index 0000000..f9b55d0
--- /dev/null
+++ b/docs/03-Plugins/github.md
@@ -0,0 +1,102 @@
+---
+title: "GitHub"
+description: >
+ GitHub Plugin
+---
+
+# Github
+
+
+## Summary
+
+This plugin gathers data from `GitHub` to display information to the user in
`Grafana`. We can help tech leaders answer such questions as:
+
+- Is this month more productive than last?
+- How fast do we respond to customer requirements?
+- Was our quality improved or not?
+
+## Metrics
+
+Here are some examples of what we can use `GitHub` data to show:
+- Avg Requirement Lead Time By Assignee
+- Bug Count per 1k Lines of Code
+- Commit Count over Time
+
+## Screenshot
+
+
+
+
+## Configuration
+
+### Provider (Datasource) Connection
+The connection aspect of the configuration screen requires the following key
fields to connect to the **GitHub API**. As GitHub is a _single-source data
provider_ at the moment, the connection name is read-only as there is only one
instance to manage. As we continue our development roadmap we may enable
_multi-source_ connections for GitHub in the future.
+
+- **Connection Name** [`READONLY`]
+ - ⚠️ Defaults to "**Github**" and may not be changed.
+- **Endpoint URL** (REST URL, starts with `https://` or `http://`)
+ - This should be a valid REST API Endpoint eg. `https://api.github.com/`
+ - ⚠️ URL should end with`/`
+- **Auth Token(s)** (Personal Access Token)
+ - For help on **Creating a personal access token**, please see official
[GitHub Docs on Personal
Tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
+ - Provide at least one token for Authentication with the . This field
accepts a comma-separated list of values for multiple tokens. The data
collection will take longer for GitHub since they have a **rate limit of 2k
requests per hour**. You can accelerate the process by configuring _multiple_
personal access tokens.
+
+"For API requests using `Basic Authentication` or `OAuth`, you can make up to
5,000 requests per hour."
+
+- https://docs.github.com/en/rest/overview/resources-in-the-rest-api
+
+If you have a need for more api rate limits, you can set many tokens in the
config file and we will use all of your tokens.
+
+NOTE: You can get 15000 requests/hour/token if you pay for `GitHub` enterprise.
+
+For an overview of the **GitHub REST API**, please see official [GitHub Docs
on REST](https://docs.github.com/en/rest)
+
+Click **Save Connection** to update connection settings.
+
+
+### Provider (Datasource) Settings
+Manage additional settings and options for the GitHub Datasource Provider.
Currently there is only one **optional** setting, *Proxy URL*. If you are
behind a corporate firewall or VPN you may need to utilize a proxy server.
+
+**GitHub Proxy URL [ `Optional`]**
+Enter a valid proxy server address on your Network, e.g.
`http://your-proxy-server.com:1080`
+
+Click **Save Settings** to update additional settings.
+
+### Regular Expression Configuration
+Define regex pattern in .env
+- GITHUB_PR_BODY_CLOSE_PATTERN: Define key word to associate issue in pr body,
please check the example in .env.example
+
+## Sample Request
+In order to collect data, you have to compose a JSON looks like following one,
and send it by selecting `Advanced Mode` on `Create Pipeline Run` page:
+```json
+[
+ [
+ {
+ "plugin": "github",
+ "options": {
+ "repo": "lake",
+ "owner": "merico-dev"
+ }
+ }
+ ]
+]
+```
+
+You can also trigger data collection by making a POST request to `/pipelines`.
+```
+curl --location --request POST 'localhost:8080/pipelines' \
+--header 'Content-Type: application/json' \
+--data-raw '
+ {
+ "name": "github 20211126",
+ "tasks": [[{
+ "plugin": "github",
+ "options": {
+ "repo": "lake",
+ "owner": "merico-dev"
+ }
+ }]]
+ }
+'
+```
+<br/><br/><br/>
diff --git a/docs/03-Plugins/gitlab.md b/docs/03-Plugins/gitlab.md
new file mode 100644
index 0000000..cec16d9
--- /dev/null
+++ b/docs/03-Plugins/gitlab.md
@@ -0,0 +1,92 @@
+---
+title: "GitLab"
+description: >
+ GitLab Plugin
+---
+
+
+## Metrics
+
+Metric Name | Description
+:------------ | :-------------
+Pull Request Count | Number of Pull/Merge Requests
+Pull Request Pass Rate | Ratio of Pull/Merge Review requests to merged
+Pull Request Reviewer Count | Number of Pull/Merge Reviewers
+Pull Request Review Time | Time from Pull/Merge created time until merged
+Commit Author Count | Number of Contributors
+Commit Count | Number of Commits
+Added Lines | Accumulated Number of New Lines
+Deleted Lines | Accumulated Number of Removed Lines
+Pull Request Review Rounds | Number of cycles of commits followed by
comments/final merge
+
+## Configuration
+
+### Provider (Datasource) Connection
+The connection aspect of the configuration screen requires the following key
fields to connect to the **GitLab API**. As GitLab is a _single-source data
provider_ at the moment, the connection name is read-only as there is only one
instance to manage. As we continue our development roadmap we may enable
_multi-source_ connections for GitLab in the future.
+
+- **Connection Name** [`READONLY`]
+ - ⚠️ Defaults to "**Gitlab**" and may not be changed.
+- **Endpoint URL** (REST URL, starts with `https://` or `http://`)
+ - This should be a valid REST API Endpoint eg.
`https://gitlab.example.com/api/v4/`
+ - ⚠️ URL should end with`/`
+- **Personal Access Token** (HTTP Basic Auth)
+ - Login to your Gitlab Account and create a **Personal Access Token** to
authenticate with the API using HTTP Basic Authentication.. The token must be
20 characters long. Save the personal access token somewhere safe. After you
leave the page, you no longer have access to the token.
+
+ 1. In the top-right corner, select your **avatar**.
+ 2. Select **Edit profile**.
+ 3. On the left sidebar, select **Access Tokens**.
+ 4. Enter a **name** and optional **expiry date** for the token.
+ 5. Select the desired **scopes**.
+ 6. Select **Create personal access token**.
+
+For help on **Creating a personal access token**, please see official [GitLab
Docs on Personal
Tokens](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html)
+
+For an overview of the **GitLab REST API**, please see official [GitLab Docs
on
REST](https://docs.gitlab.com/ee/development/documentation/restful_api_styleguide.html#restful-api)
+
+Click **Save Connection** to update connection settings.
+
+### Provider (Datasource) Settings
+There are no additional settings for the GitLab Datasource Provider at this
time.
+NOTE: `GitLab Project ID` Mappings feature has been deprecated.
+
+## Gathering Data with Gitlab
+
+To collect data, you can make a POST request to `/pipelines`
+
+```
+curl --location --request POST 'localhost:8080/pipelines' \
+--header 'Content-Type: application/json' \
+--data-raw '
+{
+ "name": "gitlab 20211126",
+ "tasks": [[{
+ "plugin": "gitlab",
+ "options": {
+ "projectId": <Your gitlab project id>
+ }
+ }]]
+}
+'
+```
+
+## Finding Project Id
+
+To get the project id for a specific `Gitlab` repository:
+- Visit the repository page on gitlab
+- Find the project id just below the title
+
+ 
+
+> Use this project id in your requests, to collect data from this project
+
+## ⚠️ (WIP) Create a Gitlab API Token <a id="gitlab-api-token"></a>
+
+1. When logged into `Gitlab` visit
`https://gitlab.com/-/profile/personal_access_tokens`
+2. Give the token any name, no expiration date and all scopes (excluding write
access)
+
+ 
+
+3. Click the **Create Personal Access Token** button
+4. Save the API token into `.env` file via `cofnig-ui` or edit the file
directly.
+
+<br/><br/><br/>
diff --git a/docs/03-Plugins/jenkins.md b/docs/03-Plugins/jenkins.md
new file mode 100644
index 0000000..a00e892
--- /dev/null
+++ b/docs/03-Plugins/jenkins.md
@@ -0,0 +1,64 @@
+---
+title: "Jenkins"
+description: >
+ Jenkins Plugin
+---
+
+# Jenkins
+
+
+## Summary
+
+This plugin collects Jenkins data through [Remote Access
API](https://www.jenkins.io/doc/book/using/remote-access-api/). It then
computes and visualizes various devops metrics from the Jenkins data.
+
+
+
+## Metrics
+
+| Metric Name | Description |
+| ------------ | ------------- |
+| Build Count | The number of builds created |
+| Build Success Rate | The percentage of successful builds |
+
+## Configuration
+
+In order to fully use this plugin, you will need to set various configurations
via Dev Lake's `config-ui`.
+
+### By `config-ui`
+
+The connection aspect of the configuration screen requires the following key
fields to connect to the Jenkins API. As Jenkins is a single-source data
provider at the moment, the connection name is read-only as there is only one
instance to manage. As we continue our development roadmap we may enable
multi-source connections for Jenkins in the future.
+
+- Connection Name [READONLY]
+ - ⚠️ Defaults to "Jenkins" and may not be changed.
+- Endpoint URL (REST URL, starts with `https://` or `http://`i, ends with `/`)
+ - This should be a valid REST API Endpoint eg. `https://ci.jenkins.io/`
+- Username (E-mail)
+ - Your User ID for the Jenkins Instance.
+- Password (Secret Phrase or API Access Token)
+ - Secret password for common credentials.
+ - For help on Username and Password, please see official Jenkins Docs on
Using Credentials
+ - Or you can use **API Access Token** for this field, which can be generated
at `User` -> `Configure` -> `API Token` section on Jenkins.
+
+Click Save Connection to update connection settings.
+
+## Collect Data From Jenkins
+
+In order to collect data from Jenkins, you have to compose a JSON looks like
following one, and send it via `Triggers` page on `config-ui`:
+
+
+```json
+[
+ [
+ {
+ "plugin": "jenkins",
+ "options": {}
+ }
+ ]
+]
+```
+
+## Relationship between job and build
+
+Build is kind of a snapshot of job. Running job each time creates a build.
+
+<br/><br/><br/>
diff --git a/docs/03-Plugins/jira.md b/docs/03-Plugins/jira.md
new file mode 100644
index 0000000..ade0a25
--- /dev/null
+++ b/docs/03-Plugins/jira.md
@@ -0,0 +1,242 @@
+---
+title: "Jira"
+description: >
+ Jira Plugin
+---
+
+
+## Summary
+
+This plugin collects Jira data through Jira Cloud REST API. It then computes
and visualizes various engineering metrics from the Jira data.
+
+<img width="2035" alt="Screen Shot 2021-09-10 at 4 01 55 PM"
src="https://user-images.githubusercontent.com/2908155/132926143-7a31d37f-22e1-487d-92a3-cf62e402e5a8.png"/>
+
+## Project Metrics This Covers
+
+| Metric Name | Description |
+| ------------| ------------ |
+| Requirement Count | Number of issues with type "Requirement" |
+| Requirement Lead Time | Lead time of issues with type "Requirement" |
+| Requirement Delivery Rate | Ratio of delivered requirements to all
requirements |
+| Requirement Granularity | Number of story points associated with an issue |
+| Bug Count | Number of issues in type "Bug". Bugs are defects found during
testing |
+| Bug Age | Lead time of issues in type "Bug". both new and deleted lines
count |
+| Bugs Count per 1k Lines of Code | Amount of bugs per 1000 lines of code |
+| Incident Count | Number of issues with type "Incident". Incidents are
defects found when running in production |
+| Incident Age | Lead time of issues with type "Incident" |
+| Incident Count per 1k Lines of Code | Amount of incidents per 1000 lines of
code |
+
+## Configuration
+
+In order to fully use this plugin, you will need to set various configurations
via Dev Lake's `config-ui` service. Open `config-ui` on browser, by default the
URL is http://localhost:4000, then go to **Data Integrations / JIRA** page.
JIRA plugin currently supports multiple data connections, Here you can **add**
new connection to your JIRA connection or **update** the settings if needed.
+
+For each connection, you will need to set up following items:
+
+- Connection Name: This allow you to distinguish different connections.
+- Endpoint URL: The JIRA instance api endpoint, for JIRA Cloud Service, it
would be: `https://{mydomain}.atlassian.net/rest`. devlake officially supports
JIRA Cloud Service on atlassian.net, may or may not work for JIRA Server
Instance.
+- Basic Auth Token: First, generate a **JIRA API TOKEN** for your JIRA account
on JIRA console (see [Generating API token](#generating-api-token)), then, in
`config-ui` click the KEY icon on the right side of the input to generate a
full `HTTP BASIC AUTH` token for you.
+- Issue Type Mapping: JIRA is highly customizable, each JIRA instance may have
a different set of issue types than others. In order to compute and visualize
metrics for different instances, you need to map your issue types to standard
ones. See [Issue Type Mapping](#issue-type-mapping) for detail.
+- Epic Key: unfortunately, epic relationship implementation in JIRA is based
on `custom field`, which is vary from instance to instance. Please see [Find
Out Custom Fields](#find-out-custom-fields).
+- Story Point Field: same as Epic Key.
+- Remotelink Commit SHA:A regular expression that matches commit links to
determine whether an external link is a link to a commit. Taking gitlab as an
example, to match all commits similar to
https://gitlab.com/merico-dev/ce/example-repository/-/commit/8ab8fb319930dbd8615830276444b8545fd0ad24,
you can directly use the regular expression **/commit/([0-9a-f]{40})$**
+
+### Generating API token
+1. Once logged into Jira, visit the url
`https://id.atlassian.com/manage-profile/security/api-tokens`
+2. Click the **Create API Token** button, and give it any label name
+
+
+
+### Issue Type Mapping
+
+Devlake supports 3 standard types, all metrics are computed based on these
types:
+
+ - `Bug`: Problems found during `test` phase, before they can reach the
production environment.
+ - `Incident`: Problems went through `test` phash, got deployed into
production environment.
+ - `Requirement`: Normally, it would be `Story` on your instance if you
adopted SCRUM.
+
+You can may map arbitrary **YOUR OWN ISSUE TYPE** to a single **STANDARD ISSUE
TYPE**, normally, one would map `Story` to `Requirement`, but you could map
both `Story` and `Task` to `Requirement` if that was your case. Those
unspecified type would be copied as standard type directly for your
convenience, so you don't need to map your `Bug` to standard `Bug`.
+
+Type mapping is critical for some metrics, like **Requirement Count**, make
sure to map your custom type correctly.
+
+### Find Out Custom Field
+
+Please follow this guide: [How to find Jira the custom field ID in
Jira?](https://github.com/apache/incubator-devlake/wiki/How-to-find-the-custom-field-ID-in-Jira)
+
+
+## Collect Data From JIRA
+
+In order to collect data from JIRA, you have to compose a JSON looks like
following one, and send it via `Triggers` page on `config-ui`.
+<font color="#ED6A45">Warning: Data collection only supports single-task
execution, and the results of concurrent multi-task execution may not meet
expectations.</font>
+
+```[
+ [
+ {
+ "plugin": "jira",
+ "options": {
+ "connectionId": 1,
+ "boardId": 8,
+ "since": "2006-01-02T15:04:05Z"
+ }
+ }
+ ]
+ ]
+```
+
+- `connectionId`: The `ID` field from **JIRA Integration** page.
+- `boardId`: JIRA board id, see "Find Board Id" for detail.
+- `since`: optional, download data since specified date/time only.
+
+
+### Find Board Id
+
+1. Navigate to the Jira board in the browser
+2. in the URL bar, get the board id from the parameter `?rapidView=`
+
+**Example:**
+
+`https://{your_jira_endpoint}/secure/RapidBoard.jspa?rapidView=51`
+
+
+
+Your board id is used in all REST requests to Apache DevLake. You do not need
to configure this at the data connection level.
+
+
+
+## API
+
+### Data Connections
+
+1. Get all data connection
+
+```GET /plugins/jira/connections
+ [
+ {
+ "ID": 14,
+ "CreatedAt": "2021-10-11T11:49:19.029Z",
+ "UpdatedAt": "2021-10-11T11:49:19.029Z",
+ "name": "test-jira-connection",
+ "endpoint": "https://merico.atlassian.net/rest";,
+ "basicAuthEncoded": "basicAuth",
+ "epicKeyField": "epicKeyField",
+ "storyPointField": "storyPointField"
+ }
+ ]
+```
+
+2. Create a new data connection
+
+```POST /plugins/jira/connections
+ {
+ "name": "jira data connection name",
+ "endpoint": "jira api endpoint, i.e. https://merico.atlassian.net/rest";,
+ "basicAuthEncoded": "generated by `echo -n {jira login email}:{jira token}
| base64`",
+ "epicKeyField": "name of customfield of epic key",
+ "storyPointField": "name of customfield of story point",
+ "typeMappings": { // optional, send empty object to delete all
typeMappings of the data connection
+ "userType": {
+ "standardType": "devlake standard type"
+ }
+ }
+ }
+```
+
+
+3. Update data connection
+
+```PUT /plugins/jira/connections/:connectionId
+ {
+ "name": "jira data connection name",
+ "endpoint": "jira api endpoint, i.e. https://merico.atlassian.net/rest";,
+ "basicAuthEncoded": "generated by `echo -n {jira login email}:{jira token}
| base64`",
+ "epicKeyField": "name of customfield of epic key",
+ "storyPointField": "name of customfield of story point",
+ "typeMappings": { // optional, send empty object to delete all
typeMappings of the data connection
+ "userType": {
+ "standardType": "devlake standard type",
+ }
+ }
+ }
+```
+
+4. Get data connection detail
+```GET /plugins/jira/connections/:connectionId
+ {
+ "name": "jira data connection name",
+ "endpoint": "jira api endpoint, i.e. https://merico.atlassian.net/rest";,
+ "basicAuthEncoded": "generated by `echo -n {jira login email}:{jira token}
| base64`",
+ "epicKeyField": "name of customfield of epic key",
+ "storyPointField": "name of customfield of story point",
+ "typeMappings": { // optional, send empty object to delete all
typeMappings of the data connection
+ "userType": {
+ "standardType": "devlake standard type",
+ }
+ }
+ }
+```
+
+5. Delete data connection
+
+```DELETE /plugins/jira/connections/:connectionId
+```
+
+
+### Type mappings
+
+1. Get all type mappings
+```GET /plugins/jira/connections/:connectionId/type-mappings
+ [
+ {
+ "jiraConnectionId": 16,
+ "userType": "userType",
+ "standardType": "standardType"
+ }
+ ]
+```
+
+2. Create a new type mapping
+
+```POST /plugins/jira/connections/:connectionId/type-mappings
+ {
+ "userType": "userType",
+ "standardType": "standardType"
+ }
+```
+
+3. Update type mapping
+
+```PUT /plugins/jira/connections/:connectionId/type-mapping/:userType
+ {
+ "standardType": "standardTypeUpdated"
+ }
+```
+
+
+4. Delete type mapping
+
+```DELETE /plugins/jira/connections/:connectionId/type-mapping/:userType
+```
+
+5. API forwarding
+For example:
+Requests to
http://your_devlake_host/plugins/jira/connections/1/proxy/rest/agile/1.0/board/8/sprint
would forward to https://your_jira_host/rest/agile/1.0/board/8/sprint
+
+```GET /plugins/jira/connections/:connectionId/proxy/rest/*path
+ {
+ "maxResults": 1,
+ "startAt": 0,
+ "isLast": false,
+ "values": [
+ {
+ "id": 7,
+ "self": "https://merico.atlassian.net/rest/agile/1.0/sprint/7";,
+ "state": "closed",
+ "name": "EE Sprint 7",
+ "startDate": "2020-06-12T00:38:51.882Z",
+ "endDate": "2020-06-26T00:38:00.000Z",
+ "completeDate": "2020-06-22T05:59:58.980Z",
+ "originBoardId": 8,
+ "goal": ""
+ }
+ ]
+ }
+```
diff --git a/docs/03-Plugins/refdiff.md b/docs/03-Plugins/refdiff.md
new file mode 100644
index 0000000..25be3ba
--- /dev/null
+++ b/docs/03-Plugins/refdiff.md
@@ -0,0 +1,115 @@
+---
+title: "RefDiff"
+description: >
+ RefDiff Plugin
+---
+
+# RefDiff
+
+
+## Summary
+
+For development workload analysis, we often need to know how many commits have
been created between 2 releases. This plugin offers the ability to calculate
the commits of difference between 2 Ref(branch/tag), and the result will be
stored back into database for further analysis.
+
+## Important Note
+
+You need to run gitextractor before the refdiff plugin. The gitextractor
plugin should create records in the `refs` table in your DB before this plugin
can be run.
+
+## Configuration
+
+This is a enrichment plugin based on Domain Layer data, no configuration needed
+
+## How to use
+
+In order to trigger the enrichment, you need to insert a new task into your
pipeline.
+
+1. Make sure `commits` and `refs` are collected into your database, `refs`
table should contain records like following:
+```
+id ref_type
+github:GithubRepo:384111310:refs/tags/0.3.5 TAG
+github:GithubRepo:384111310:refs/tags/0.3.6 TAG
+github:GithubRepo:384111310:refs/tags/0.5.0 TAG
+github:GithubRepo:384111310:refs/tags/v0.0.1 TAG
+github:GithubRepo:384111310:refs/tags/v0.2.0 TAG
+github:GithubRepo:384111310:refs/tags/v0.3.0 TAG
+github:GithubRepo:384111310:refs/tags/v0.4.0 TAG
+github:GithubRepo:384111310:refs/tags/v0.6.0 TAG
+github:GithubRepo:384111310:refs/tags/v0.6.1 TAG
+```
+2. If you want to run calculateIssuesDiff, please configure
GITHUB_PR_BODY_CLOSE_PATTERN in .env, you can check the example in
.env.example(we have a default value, please make sure your pattern is
disclosed by single quotes '')
+3. If you want to run calculatePrCherryPick, please configure
GITHUB_PR_TITLE_PATTERN in .env, you can check the example in .env.example(we
have a default value, please make sure your pattern is disclosed by single
quotes '')
+4. And then, trigger a pipeline like following, you can also define sub tasks,
calculateRefDiff will calculate commits between two ref, and creatRefBugStats
will create a table to show bug list between two ref:
+```
+curl -v -XPOST http://localhost:8080/pipelines --data @- <<'JSON'
+{
+ "name": "test-refdiff",
+ "tasks": [
+ [
+ {
+ "plugin": "refdiff",
+ "options": {
+ "repoId": "github:GithubRepo:384111310",
+ "pairs": [
+ { "newRef": "refs/tags/v0.6.0", "oldRef":
"refs/tags/0.5.0" },
+ { "newRef": "refs/tags/0.5.0", "oldRef":
"refs/tags/0.4.0" }
+ ],
+ "tasks": [
+ "calculateCommitsDiff",
+ "calculateIssuesDiff",
+ "calculatePrCherryPick",
+ ]
+ }
+ }
+ ]
+ ]
+}
+JSON
+```
+
+## Development
+
+This plugin depends on `libgit2`, you need to install version 1.3.0 in order
to run and debug this plugin on your local
+machine.
+
+### Ubuntu
+
+```
+apt install cmake
+git clone https://github.com/libgit2/libgit2.git
+git checkout v1.3.0
+mkdir build
+cd build
+cmake ..
+make
+make install
+```
+
+### MacOs
+
+```
+brew install cmake
+git clone https://github.com/libgit2/libgit2.git
+cd libgit2
+git checkout v1.3.0
+mkdir build
+cd build
+cmake ..
+make
+make install
+```
+
+Troubleshooting (MacOS)
+
+Q: I got an error saying: `pkg-config: exec: "pkg-config": executable file not
found in $PATH`
+
+A:
+
+1. Make sure you have pkg-config installed:
+
+ `brew install pkg-config`
+
+2. Make sure your pkg config path covers the installation:
+
+ `export
PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib:/usr/local/lib/pkgconfig`
+
+<br/><br/><br/>
diff --git a/docs/04-UserManuals/GRAFANA.md b/docs/04-UserManuals/GRAFANA.md
new file mode 100644
index 0000000..5837c14
--- /dev/null
+++ b/docs/04-UserManuals/GRAFANA.md
@@ -0,0 +1,119 @@
+---
+title: "How to use Grafana"
+description: >
+ How to use Grafana
+---
+
+
+# Grafana
+
+<img
src="https://user-images.githubusercontent.com/3789273/128533901-3107e9bf-c3e3-4320-ba47-879fe2b0ea4d.png";
width="450px" />
+
+When first visiting grafana, you will be provided with a sample dashboard with
some basic charts setup from the database
+
+## Contents
+
+Section | Link
+:------------ | :-------------
+Logging In | [View Section](#logging-in)
+Viewing All Dashboards | [View Section](#viewing-all-dashboards)
+Customizing a Dashboard | [View Section](#customizing-a-dashboard)
+Dashboard Settings | [View Section](#dashboard-settings)
+Provisioning a Dashboard | [View Section](#provisioning-a-dashboard)
+Troubleshooting DB Connection | [View Section](#troubleshooting-db-connection)
+
+## Logging In<a id="logging-in"></a>
+
+Once the app is up and running, visit `http://localhost:3002` to view the
Grafana dashboard.
+
+Default login credentials are:
+
+- Username: `admin`
+- Password: `admin`
+
+## Viewing All Dashboards<a id="viewing-all-dashboards"></a>
+
+To see all dashboards created in Grafana visit `/dashboards`
+
+Or, use the sidebar and click on **Manage**:
+
+
+
+
+## Customizing a Dashboard<a id="customizing-a-dashboard"></a>
+
+When viewing a dashboard, click the top bar of a panel, and go to **edit**
+
+
+
+**Edit Dashboard Panel Page:**
+
+
+
+### 1. Preview Area
+- **Top Left** is the variable select area (custom dashboard variables, used
for switching projects, or grouping data)
+- **Top Right** we have a toolbar with some buttons related to the display of
the data:
+ - View data results in a table
+ - Time range selector
+ - Refresh data button
+- **The Main Area** will display the chart and should update in real time
+
+> Note: Data should refresh automatically, but may require a refresh using the
button in some cases
+
+### 2. Query Builder
+Here we form the SQL query to pull data into our chart, from our database
+- Ensure the **Data Source** is the correct database
+
+ 
+
+- Select **Format as Table**, and **Edit SQL** buttons to write/edit queries
as SQL
+
+ 
+
+- The **Main Area** is where the queries are written, and in the top right is
the **Query Inspector** button (to inspect returned data)
+
+ 
+
+### 3. Main Panel Toolbar
+In the top right of the window are buttons for:
+- Dashboard settings (regarding entire dashboard)
+- Save/apply changes (to specific panel)
+
+### 4. Grafana Parameter Sidebar
+- Change chart style (bar/line/pie chart etc)
+- Edit legends, chart parameters
+- Modify chart styling
+- Other Grafana specific settings
+
+## Dashboard Settings<a id="dashboard-settings"></a>
+
+When viewing a dashboard click on the settings icon to view dashboard
settings. In here there is 2 pages important sections to use:
+
+
+
+- Variables
+ - Create variables to use throughout the dashboard panels, that are also
built on SQL queries
+
+ 
+
+- JSON Model
+ - Copy `json` code here and save it to a new file in `/grafana/dashboards/`
with a unique name in the `lake` repo. This will allow us to persist dashboards
when we load the app
+
+ 
+
+## Provisioning a Dashboard<a id="provisioning-a-dashboard"></a>
+
+To save a dashboard in the `lake` repo and load it:
+
+1. Create a dashboard in browser (visit `/dashboard/new`, or use sidebar)
+2. Save dashboard (in top right of screen)
+3. Go to dashboard settings (in top right of screen)
+4. Click on _JSON Model_ in sidebar
+5. Copy code into a new `.json` file in `/grafana/dashboards`
+
+## Troubleshooting DB Connection<a id="troubleshooting-db-connection"></a>
+
+To ensure we have properly connected our database to the data source in
Grafana, check database settings in `./grafana/datasources/datasource.yml`,
specifically:
+- `database`
+- `user`
+- `secureJsonData/password`
diff --git a/docs/04-UserManuals/MIGRATIONS.md
b/docs/04-UserManuals/MIGRATIONS.md
new file mode 100644
index 0000000..4d3f1e2
--- /dev/null
+++ b/docs/04-UserManuals/MIGRATIONS.md
@@ -0,0 +1,36 @@
+---
+title: "DB Migration"
+description: >
+ DB Migration
+---
+
+# Migrations (Database)
+
+## Summary
+Starting in v0.10.0, DevLake provides a lightweight migration tool for
executing migration scripts.
+Both framework itself and plugins define their migration scripts in their own
migration folder.
+The migration scripts are written with gorm in Golang to support different SQL
dialects.
+
+
+## Migration script
+Migration script describes how to do database migration.
+They implement the `Script` interface.
+When DevLake starts, scripts register themselves to the framework by invoking
the `Register` function
+
+```go
+type Script interface {
+ Up(ctx context.Context, db *gorm.DB) error
+ Version() uint64
+ Name() string
+}
+```
+
+## Table `migration_history`
+
+The table tracks migration scripts execution and schemas changes.
+From which, DevLake could figure out the current state of database schemas.
+## How it Works
+1. check `migration_history` table, calculate all the migration scripts need
to be executed.
+2. sort scripts by Version in ascending order.
+3. execute scripts.
+4. save results in the `migration_history` table.
diff --git a/docs/04-UserManuals/NOTIFICATION.md
b/docs/04-UserManuals/NOTIFICATION.md
new file mode 100644
index 0000000..8d7b855
--- /dev/null
+++ b/docs/04-UserManuals/NOTIFICATION.md
@@ -0,0 +1,33 @@
+---
+title: "Notifications"
+description: >
+ Notifications
+---
+
+# Notification
+
+## Request
+example request
+```
+POST
/lake/notify?nouce=3-FDXxIootApWxEVtz&sign=424c2f6159bd9e9828924a53f9911059433dc14328a031e91f9802f062b495d5
+
+{"TaskID":39,"PluginName":"jenkins","CreatedAt":"2021-09-30T15:28:00.389+08:00","UpdatedAt":"2021-09-30T15:28:00.785+08:00"}
+```
+
+## Configuration
+If you want to use the notification feature, you should add two configuration
key to `.env` file.
+```shell
+# .env
+# endpoint is the notification request url, eg: http://example.com/lake/notify
+NOTIFICATION_ENDPOINT=
+# screte is used to calculate signature
+NOTIFICATION_SECRET=
+```
+
+## Signature
+You should check the signature before accepting the notification request. We
use sha256 algorithm to calculate the checksum.
+```go
+// calculate checksum
+sum := sha256.Sum256([]byte(requestBody + NOTIFICATION_SECRET + nouce))
+return hex.EncodeToString(sum[:])
+```
diff --git a/docs/04-UserManuals/_category_.json
b/docs/04-UserManuals/_category_.json
new file mode 100644
index 0000000..60bb981
--- /dev/null
+++ b/docs/04-UserManuals/_category_.json
@@ -0,0 +1,4 @@
+{
+ "label": "User Manuals",
+ "position": 4
+}
diff --git a/docs/04-UserManuals/create-pipeline-in-advanced-mode.md
b/docs/04-UserManuals/create-pipeline-in-advanced-mode.md
new file mode 100644
index 0000000..1f64054
--- /dev/null
+++ b/docs/04-UserManuals/create-pipeline-in-advanced-mode.md
@@ -0,0 +1,88 @@
+---
+title: "Create Pipeline in Advanced Mode"
+description: >
+ Create Pipeline in Advanced Mode
+---
+
+
+## Why advanced mode?
+
+Advanced mode allows users to create any pipeline by writing JSON. This is
most useful for users who'd like to:
+
+1. Collect multiple GitHub/GitLab repos or Jira projects within a single
pipeline
+2. Have fine-grained control over what entities to collect or what subtasks to
run for each plugin
+3. Orchestrate a complex pipeline that consists of multiple stages of plugins.
+
+Advaned mode gives the most flexiblity to users by exposing the JSON API
+
+## How to use advanced mode to create pipelines?
+
+1. Visit the "Create Pipeline Run" page on `config-ui`
+
+
+
+2. Scroll to the bottom and toggle on the "Advanced Mode" button
+
+
+
+3. The pipeline editor expects a 2D array of plugins. The first dimension
represents different stages of the pipeline and the second dimension describes
the plugins in each stage. Stages run in sequential order and plugins within
the same stage runs in parallel. We provide some templates for users to get
started. Please also see the next section for some examples.
+
+
+
+## Examples
+
+1. Collect multiple GitLab repos sequentially.
+
+>When there're multiple collection tasks against a single data source, we
recommend running these tasks sequentially since the collection speed is mostly
limited by the API rate limit of the data source.
+>Running multiple tasks against the same data source is unlikely to speed up
the process and may overwhelm the data source.
+
+
+Below is an example for collecting 2 GitLab repos sequentially. It has 2
stages, each contains a GitLab task.
+
+
+```
+[
+ [
+ {
+ "Plugin": "gitlab",
+ "Options": {
+ "projectId": 15238074
+ }
+ }
+ ],
+ [
+ {
+ "Plugin": "gitlab",
+ "Options": {
+ "projectId": 11624398
+ }
+ }
+ ]
+]
+```
+
+
+2. Collect a GitHub repo and a Jira board in parallel
+
+Below is an example for collecting a GitHub repo and a Jira board in parallel.
It has a single stage with a GitHub task and a Jira task. Since users can
configure multiple Jira connection, it's required to pass in a `connectionId`
for Jira task to specify which connection to use.
+
+```
+[
+ [
+ {
+ "Plugin": "github",
+ "Options": {
+ "repo": "lake",
+ "owner": "merico-dev"
+ }
+ },
+ {
+ "Plugin": "jira",
+ "Options": {
+ "connectionId": 1,
+ "boardId": 76
+ }
+ }
+ ]
+]
+```
diff --git a/docs/04-UserManuals/github-user-guide-v0.10.0.md
b/docs/04-UserManuals/github-user-guide-v0.10.0.md
new file mode 100644
index 0000000..298d62c
--- /dev/null
+++ b/docs/04-UserManuals/github-user-guide-v0.10.0.md
@@ -0,0 +1,119 @@
+---
+title: "GitHub User Guide v0.10.0"
+description: >
+ GitHub User Guide v0.10.0
+---
+
+## Summary
+
+GitHub has a rate limit of 2,000 API calls per hour for their REST API.
+As a result, it may take hours to collect commits data from GitHub API for a
repo that has 10,000+ commits.
+To accelerate the process, DevLake introduces GitExtractor, a new plugin that
collects git data by cloning the git repo instead of by calling GitHub APIs.
+
+Starting from v0.10.0, DevLake will collect GitHub data in 2 separate plugins:
+
+- GitHub plugin (via GitHub API): collect repos, issues, pull requests
+- GitExtractor (via cloning repos): collect commits, refs
+
+Note that GitLab plugin still collects commits via API by default since GitLab
has a much higher API rate limit.
+
+This doc details the process of collecting GitHub data in v0.10.0. We're
working on simplifying this process in the next releases.
+
+Before start, please make sure all services are started.
+
+## GitHub Data Collection Procedure
+
+There're 3 steps.
+
+1. Configure GitHub connection
+2. Create a pipeline to run GitHub plugin
+3. Create a pipeline to run GitExtractor plugin
+4. [Optional] Set up a recurring pipeline to keep data fresh
+
+### Step 1 - Configure GitHub connection
+
+1. Visit `config-ui` at `http://localhost:4000`, click the GitHub icon
+
+2. Click the default connection 'Github' in the list
+
+
+3. Configure connection by providing your GitHub API endpoint URL and your
personal access token(s).
+
+
+ > Endpoint URL: Leave this unchanged if you're using github.com. Otherwise
replace it with your own GitHub instance's REST API endpoint URL. This URL
should end with '/'.
+ >
+ > Auth Token(s): Fill in your personal access tokens(s). For how to
generate personal access tokens, please see GitHub's [official
documentation](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token).
+ > You can provide multiple tokens to speed up the data collection process,
simply concatenating tokens with commas.
+ >
+ > GitHub Proxy URL: This is optional. Enter a valid proxy server address
on your Network, e.g. http://your-proxy-server.com:1080
+
+4. Click 'Test Connection' and see it's working, then click 'Save Connection'.
+
+5. [Optional] Help DevLake understand your GitHub data by customizing data
enrichment rules shown below.
+
+
+ 1. Pull Request Enrichment Options
+
+ 1. `Type`: PRs with label that matches given Regular Expression, their
properties `type` will be set to the value of first sub match. For example,
with Type being set to `type/(.*)$`, a PR with label `type/bug`, its `type`
would be set to `bug`, with label `type/doc`, it would be `doc`.
+ 2. `Component`: Same as above, but for `component` property.
+
+ 2. Issue Enrichment Options
+
+ 1. `Severity`: Same as above, but for `issue.severity` of course.
+
+ 2. `Component`: Same as above.
+
+ 3. `Priority`: Same as above.
+
+ 4. **Requirement** : Issues with label that matches given Regular
Expression, their properties `type` will be set to `REQUIREMENT`. Unlike
`PR.type`, submatch does nothing, because for Issue Management Analysis,
people tend to focus on 3 kinds of type (Requiremnt/Bug/Incident), however, the
concrete naming varies from repo to repo, time to time, so we decided to
standardize them to help analyst making general purpose metric.
+
+ 5. **Bug**: Same as above, with `type` setting to `BUG`
+
+ 6. **Incident**: Same as above, with `type` setting to `INCIDENT`
+
+6. Click 'Save Settings'
+
+### Step 2 - Create a pipeline to collect GitHub data
+
+1. Select 'Pipelines > Create Pipeline Run' from `config-ui`
+
+
+
+2. Toggle on GitHub plugin, enter the repo you'd like to collect data from.
+
+
+
+3. Click 'Run Pipeline'
+
+You'll be redirected to newly created pipeline:
+
+
+
+
+See the pipeline finishes (progress 100%):
+
+
+
+### Step 3 - Create a pipeline to run GitExtractor plugin
+
+1. Enable the `GitExtractor` plugin, and enter your `Git URL` and, select the
`Repository ID` from dropdown menu.
+
+
+
+2. Click 'Run Pipeline' and wait until it's finished.
+
+3. Click `View Dashboards` on the top left corner of `config-ui`
+
+
+
+4. See dashboards populated with GitHub data.
+
+### Step 4 - [Optional] Set up a recurring pipeline to keep data fresh
+
+Please see [How to create recurring pipelines](./recurring-pipeline.md) for
details.
+
+
+
+
+
+
diff --git a/docs/04-UserManuals/recurring-pipeline.md
b/docs/04-UserManuals/recurring-pipeline.md
new file mode 100644
index 0000000..29f3698
--- /dev/null
+++ b/docs/04-UserManuals/recurring-pipeline.md
@@ -0,0 +1,29 @@
+---
+title: "Create Recurring Pipelines"
+description: >
+ Create Recurring Pipelines
+---
+
+## How to create recurring pipelines?
+
+Once you've verified a pipeline works well, mostly likely you'll want to run
that pipeline periodically to keep data fresh, and DevLake's pipeline blueprint
feature have got you covered.
+
+
+1. Click 'Create Pipeline Run' and
+ - Toggle the plugins you'd like to run, here we use GitHub and GitExtractor
plugin as an example
+ - Toggle on Automate Pipeline
+
+
+
+2. Click 'Add Blueprint'. Fill in the form and 'Save Blueprint'.
+
+ - **NOTE**: That the schedule syntax is standard unix cron syntax,
[Crontab.guru](https://crontab.guru/) could be a useful reference
+ - **IMPORANT**: The scheduler is running under `UTC` timezone. If you
prefer data collecting happens at 3am NewYork(UTC-04:00) every day, use
**Custom Shedule** and set it to `0 7 * * *`
+
+
+
+3. Click 'Save Blueprint'.
+
+4. Click 'Pipeline Blueprints', you can view and edit the new blueprint in the
blueprint list.
+
+
\ No newline at end of file
diff --git a/docs/04-DataModels/01-DevLakeDomainLayerSchema.md
b/docs/05-DataModels/01-DevLakeDomainLayerSchema.md
similarity index 100%
rename from docs/04-DataModels/01-DevLakeDomainLayerSchema.md
rename to docs/05-DataModels/01-DevLakeDomainLayerSchema.md
diff --git a/docs/04-DataModels/02-DataSupport.md
b/docs/05-DataModels/02-DataSupport.md
similarity index 100%
rename from docs/04-DataModels/02-DataSupport.md
rename to docs/05-DataModels/02-DataSupport.md
diff --git a/docs/04-DataModels/_category_.json
b/docs/05-DataModels/_category_.json
similarity index 65%
rename from docs/04-DataModels/_category_.json
rename to docs/05-DataModels/_category_.json
index 7c004be..e678e71 100644
--- a/docs/04-DataModels/_category_.json
+++ b/docs/05-DataModels/_category_.json
@@ -1,4 +1,4 @@
{
"label": "Data Models",
- "position": 4
+ "position": 5
}
diff --git a/docs/05-EngineeringMetrics.md b/docs/06-EngineeringMetrics.md
similarity index 99%
rename from docs/05-EngineeringMetrics.md
rename to docs/06-EngineeringMetrics.md
index bc98a76..54debf3 100644
--- a/docs/05-EngineeringMetrics.md
+++ b/docs/06-EngineeringMetrics.md
@@ -1,5 +1,5 @@
---
-sidebar_position: 05
+sidebar_position: 06
title: "Engineering Metrics"
linkTitle: "Engineering Metrics"
tags: []
diff --git a/docs/06-Glossary.md b/docs/07-Glossary.md
similarity index 88%
rename from docs/06-Glossary.md
rename to docs/07-Glossary.md
index 7972458..79b01f4 100644
--- a/docs/06-Glossary.md
+++ b/docs/07-Glossary.md
@@ -1,5 +1,5 @@
---
-sidebar_position: 06
+sidebar_position: 07
title: "Glossary"
linkTitle: "Glossary"
tags: []
@@ -18,9 +18,9 @@ The following terms are arranged in the order of their
appearance in the actual
### Blueprints
**A blueprint is the plan that covers all the work to get your raw data ready
for query and metric computaion in the dashboards.** Creating a blueprint
consists of four steps:
-1. **Adding [Data Connections](06-Glossary.md#data-connections)**: For each
[data source](06-Glossary.md#data-sources), one or more data connections can be
added to a single blueprint, depending on the data you want to sync to DevLake.
-2. **Setting the [Data Scope](06-Glossary.md#data-scope)**: For each data
connection, you need to configure the scope of data, such as GitHub projects,
Jira boards, and their corresponding [data
entities](06-Glossary.md#data-entities).
-3. **Adding [Transformation Rules](06-Glossary.md#transformation-rules)
(optional)**: You can optionally apply transformation for the data scope you
have just selected, in order to view more advanced metrics.
+1. **Adding [Data Connections](07-Glossary.md#data-connections)**: For each
[data source](07-Glossary.md#data-sources), one or more data connections can be
added to a single blueprint, depending on the data you want to sync to DevLake.
+2. **Setting the [Data Scope](07-Glossary.md#data-scope)**: For each data
connection, you need to configure the scope of data, such as GitHub projects,
Jira boards, and their corresponding [data
entities](07-Glossary.md#data-entities).
+3. **Adding [Transformation Rules](07-Glossary.md#transformation-rules)
(optional)**: You can optionally apply transformation for the data scope you
have just selected, in order to view more advanced metrics.
3. **Setting the Sync Frequency**: You can specify the sync frequency for your
blueprint to achieve recurring data syncs and transformation. Alternatively,
you can set the frequency to manual if you wish to run the tasks in the
blueprint manually.
The relationship among Blueprint, Data Connections, Data Scope and
Transformation Rules is explained as follows:
@@ -34,7 +34,7 @@ The relationship among Blueprint, Data Connections, Data
Scope and Transformatio
### Data Sources
**A data source is a specific DevOps tool from which you wish to sync your
data, such as GitHub, GitLab, Jira and Jenkins.**
-DevLake normally uses one [data plugin](06-Glossary.md#data-plugins) to pull
data for a single data source. However, in some cases, DevLake uses multiple
data plugins for one data source for the purpose of improved sync speed, among
many other advantages. For instance, when you pull data from GitHub or GitLab,
aside from the GitHub or GitLab plugin, Git Extractor is also used to pull data
from the repositories. In this case, DevLake still refers GitHub or GitLab as a
single data source.
+DevLake normally uses one [data plugin](07-Glossary.md#data-plugins) to pull
data for a single data source. However, in some cases, DevLake uses multiple
data plugins for one data source for the purpose of improved sync speed, among
many other advantages. For instance, when you pull data from GitHub or GitLab,
aside from the GitHub or GitLab plugin, Git Extractor is also used to pull data
from the repositories. In this case, DevLake still refers GitHub or GitLab as a
single data source.
### Data Connections
**A data connection is a specific instance of a data source that stores
information such as `endpoint` and `auth`.** A single data source can have one
or more data connections (e.g. two Jira instances). Currently, DevLake supports
one data connection for GitHub, GitLab and Jenkins, and multiple connections
for Jira.
@@ -42,18 +42,18 @@ DevLake normally uses one [data
plugin](06-Glossary.md#data-plugins) to pull dat
You can set up a new data connection either during the first step of creating
a blueprint, or in the Connections page that can be accessed from the
navigation bar. Because one single data connection can be resued in multiple
blueprints, you can update the information of a particular data connection in
Connections, to ensure all its associated blueprints will run properly. For
example, you may want to update your GitHub token in a data connection if it
goes expired.
### Data Scope
-**In a blueprint, each data connection can have multiple sets of data scope
configurations, including GitHub or GitLab projects, Jira boards and their
corresponding[data entities](06-Glossary.md#data-entities).** The fields for
data scope configuration vary according to different data sources.
+**In a blueprint, each data connection can have multiple sets of data scope
configurations, including GitHub or GitLab projects, Jira boards and their
corresponding[data entities](07-Glossary.md#data-entities).** The fields for
data scope configuration vary according to different data sources.
Each set of data scope refers to one GitHub or GitLab project, or one Jira
board and the data entities you would like to sync for them, for the
convinience of applying transformation in the next step. For instance, if you
wish to sync 5 GitHub projects, you will have 5 sets of data scope for GitHub.
-To learn more about the default data scope of all data sources and data
plugins, please refer to [Data Support](04-DataModels/02-DataSupport.md).
+To learn more about the default data scope of all data sources and data
plugins, please refer to [Data Support](./05-DataModels/02-DataSupport.md).
### Data Entities
**Data entities refer to the data fields from one of the five data domains:
Issue Tracking, Source Code Management, Code Review, CI/CD and Cross-Domain.**
For instance, if you wish to pull Source Code Management data from GitHub and
Issue Tracking data from Jira, you can check the corresponding data entities
during setting the data scope of these two data connections.
-To learn more details, please refer to [Domain Layer
Schema](04-DataModels/01-DevLakeDomainLayerSchema.md).
+To learn more details, please refer to [Domain Layer
Schema](./05-DataModels/01-DevLakeDomainLayerSchema.md).
### Transformation Rules
**Transformation rules are a collection of methods that allow you to customize
how DevLake normalizes raw data for query and metric computation.** Each set of
data scope is strictly acompanied with one set of transformation rules.
However, for your convenience, transformation rules can also be duplicated
across different sets of data scope.
@@ -61,12 +61,12 @@ To learn more details, please refer to [Domain Layer
Schema](04-DataModels/01-De
DevLake uses these normalized values in the transformtion to design more
advanced dashboards, such as the Weekly Bug Retro dashboard. Although
configuring transformation rules is not mandatory, if you leave the rules blank
or have not configured correctly, only the basic dashboards (e.g. GitHub Basic
Metrics) will be displayed as expected, while the advanced dashboards will not.
### Historical Runs
-**A historical run of a blueprint is an actual excecution of the data
collection and transformation [tasks](06-Glossary.md#tasks) defined in the
blueprint at its creation.** A list of historical runs of a blueprint is the
entire running history of that blueprint, whether excecuted automatically or
manually. Historical runs can be triggered in three ways:
+**A historical run of a blueprint is an actual excecution of the data
collection and transformation [tasks](07-Glossary.md#tasks) defined in the
blueprint at its creation.** A list of historical runs of a blueprint is the
entire running history of that blueprint, whether excecuted automatically or
manually. Historical runs can be triggered in three ways:
- By the blueprint automatically according to its schedule in the Regular Mode
of the Configuration UI
- By running the JSON in the Advanced Mode of the Configuration UI
- By calling the API `/pipelines` endpoint manually
-However, the name Historical Runs is only used in the Configuration UI. In
DevLake API, they are called [pipelines](06-Glossary.md#pipelines).
+However, the name Historical Runs is only used in the Configuration UI. In
DevLake API, they are called [pipelines](07-Glossary.md#pipelines).
## In Configuration UI (Advanced Mode) and API
@@ -81,11 +81,11 @@ Data Transformation Plugins transform the data pulled by
other Data Collection P
Although the names of the data plugins are not displayed in the regular mode
of DevLake Configuration UI, they can be used directly in JSON in the Advanced
Mode.
-For detailed information about the relationship between data sources and data
plugins, please refer to [Data Support](04-DataModels/02-DataSupport.md).
+For detailed information about the relationship between data sources and data
plugins, please refer to [Data Support](./05-DataModels/02-DataSupport.md).
### Pipelines
-**A pipeline is an orchestration of [tasks](06-Glossary.md#tasks) of data
`collection`, `extraction`, `conversion` and `enrichment`, defined in the
DevLake API.** A pipeline is composed of one or multiple
[stages](06-Glossary.md#stages) that are executed in a sequential order. Any
error occured during the execution of any stage, task or substask will cause
the immediate fail of the pipeline.
+**A pipeline is an orchestration of [tasks](07-Glossary.md#tasks) of data
`collection`, `extraction`, `conversion` and `enrichment`, defined in the
DevLake API.** A pipeline is composed of one or multiple
[stages](07-Glossary.md#stages) that are executed in a sequential order. Any
error occured during the execution of any stage, task or substask will cause
the immediate fail of the pipeline.
The composition of a pipeline is exaplined as follows:
@@ -95,7 +95,7 @@ Notice: **You can manually orchestrate the pipeline in
Configuration UI Advanced
**A stages is a collection of tasks performed by data plugins.** Stages are
executed in a sequential order in a pipeline.
### Tasks
-**A task is a collection of [subtasks](06-Glossary.md#subtasks) that perform
any of the `collection`, `extraction`, `conversion` and `enrichment` jobs of a
particular data plugin.** Tasks are executed in a parralel order in any stages.
+**A task is a collection of [subtasks](07-Glossary.md#subtasks) that perform
any of the `collection`, `extraction`, `conversion` and `enrichment` jobs of a
particular data plugin.** Tasks are executed in a parralel order in any stages.
### Subtasks
**A subtask is the minimal work unit in a pipeline that performs in any of the
four roles: `Collectors`, `Extractors`, `Converters` and `Enrichers`.**
Subtasks are executed in sequential orders.
diff --git a/docs/07-LiveDemo.md b/docs/08-LiveDemo.md
similarity index 94%
rename from docs/07-LiveDemo.md
rename to docs/08-LiveDemo.md
index 49a98fa..23963d7 100644
--- a/docs/07-LiveDemo.md
+++ b/docs/08-LiveDemo.md
@@ -1,5 +1,5 @@
---
-sidebar_position: 07
+sidebar_position: 08
title: "Dashboards (Live Demo)"
linkTitle: "Live Demo"
tags: []
diff --git a/static/img/github-demo.png b/static/img/github-demo.png
new file mode 100644
index 0000000..ba31588
Binary files /dev/null and b/static/img/github-demo.png differ
diff --git a/static/img/jenkins-demo.png b/static/img/jenkins-demo.png
new file mode 100644
index 0000000..a27b5c3
Binary files /dev/null and b/static/img/jenkins-demo.png differ
diff --git a/static/img/jira-demo.png b/static/img/jira-demo.png
new file mode 100644
index 0000000..c85b553
Binary files /dev/null and b/static/img/jira-demo.png differ
![https://user-images.githubusercontent.com/2908155/132926143-7a31d37f-22e1-487d-92a3-cf62e402e5a8.png"/](https://user-images.githubusercontent.com/2908155/132926143-7a31d37f-22e1-487d-92a3-cf62e402e5a8.png"/){kind=link}
![https://user-images.githubusercontent.com/3789273/128533901-3107e9bf-c3e3-4320-ba47-879fe2b0ea4d.png"](https://user-images.githubusercontent.com/3789273/128533901-3107e9bf-c3e3-4320-ba47-879fe2b0ea4d.png"){kind=link}
