This is an automated email from the ASF dual-hosted git repository. zky pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-devlake-website.git
The following commit(s) were added to refs/heads/main by this push:
new da9259069 docs: updated supported-data-sources-v0.15.0 (#364)
da9259069 is described below
commit da925906928f8e1a32e559ce71e76432fe8a91ef
Author: abeizn <[email protected]>
AuthorDate: Wed Dec 28 14:16:28 2022 +0800
docs: updated supported-data-sources-v0.15.0 (#364)
* docs: updated supported-data-sources-v0.15.0
* docs: updated supported-data-sources-v0.15.0
* docs: updated supported-data-sources-v0.15.0
* docs: updated supported-data-sources-v0.15.0
---
docs/Overview/SupportedDataSources.md | 224 +++++++++++++++++-----------------
1 file changed, 114 insertions(+), 110 deletions(-)
diff --git a/docs/Overview/SupportedDataSources.md
b/docs/Overview/SupportedDataSources.md
index 39dacf44c..1c41770a5 100644
--- a/docs/Overview/SupportedDataSources.md
+++ b/docs/Overview/SupportedDataSources.md
@@ -5,61 +5,77 @@ description: >
sidebar_position: 5
---
-
## Data Sources and Data Plugins
+
Apache DevLake(incubating) supports the following data sources. The data from
each data source is collected with one or more plugins.
-| Data Source | Domain(s)
| Supported Versions | Config UI Availability
| Triggered Plugins | Collection Mode
|
-| ------------------------------ |
---------------------------------------------------------- |
------------------------------------ | ---------------------- |
--------------------------- |
------------------------------------------------------------ |
-| GitHub (include GitHub Action) | Source Code Management, Code Review, Issue
Tracking, CI/CD | Cloud | Available
| `github`, `gitextractor` | Full Refresh
|
-| GitLab (include GitLabCI) | Source Code Management, Code Review, Issue
Tracking, CI/CD | Cloud, Community Edition 13.x+ | Available
| `gitlab`, `gitextractor` | Full Refresh, Incremental Sync(for
`issues`,`MRs`) |
-| Gitee | Source Code Management, Code Review, Issue
Tracking | Cloud | Not Available
| `gitee`, `gitextractor` | Full Refresh, Incremental Sync(for
`issues`,`MRs`) |
-| BitBucket | Source Code Management, Code Review
| Cloud | Not Available
| `bitbucket`, `gitextractor` | Full Refresh
|
-| Jira | Issue Tracking
| Cloud, Server 7.x+, Data Center 7.x+ | Available
| `jira` | Full Refresh, Incremental Sync(for `issues` and
related) |
+| Data Source | Domain(s)
| Supported Versions | Config UI Availability
| Triggered Plugins | Collection Mode
|
+| ------------------------------ |
---------------------------------------------------------- |
------------------------------------ | ---------------------- |
--------------------------- |
-------------------------------------------------------------- |
+| GitHub (include GitHub Action) | Source Code Management, Code Review, Issue
Tracking, CI/CD | Cloud | Available
| `github`, `gitextractor` | Full Refresh
|
+| GitLab (include GitLabCI) | Source Code Management, Code Review, Issue
Tracking, CI/CD | Cloud, Community Edition 13.x+ | Available
| `gitlab`, `gitextractor` | Full Refresh, Incremental Sync(for
`issues`,`MRs`) |
+| Gitee | Source Code Management, Code Review, Issue
Tracking | Cloud | Not Available
| `gitee`, `gitextractor` | Full Refresh, Incremental Sync(for
`issues`,`MRs`) |
+| BitBucket | Source Code Management, Code Review
| Cloud | Not Available
| `bitbucket`, `gitextractor` | Full Refresh
|
+| Jira | Issue Tracking
| Cloud, Server 7.x+, Data Center 7.x+ | Available
| `jira` | Full Refresh, Incremental Sync(for `issues` and
related) |
| TAPD | Issue Tracking
| Cloud | Not Available
| `tapd` | Full Refresh, Incremental Sync(for `stories`,
`bugs`, `tasks`) |
-| Jenkins | CI/CD
| 2.263.x+ | Available
| `jenkins` | Full Refresh
|
-| Feishu | Calendar
| Cloud | Not Available
| `feishu` | Full Refresh
|
-| AE | Source Code Management
| | Not Available
| `ae` | Full Refresh
|
-
-
+| Jenkins | CI/CD
| 2.263.x+ | Available
| `jenkins` | Full Refresh
|
+| Feishu | Calendar
| Cloud | Not Available
| `feishu` | Full Refresh
|
+| AE | Source Code Management
| | Not Available
| `ae` | Full Refresh
|
## Data Collection Scope By Each Plugin
-This table shows the entities collected by each plugin. Domain layer entities
in this table are consistent with the entities
[here](/DataModels/DevLakeDomainLayerSchema.md).
-
-| Domain Layer Entities | ae | gitextractor | github |
gitlab | jenkins | jira | refdiff | tapd |
-| --------------------- | -------------- | ------------ | -------------- |
------- | ------- | ------- | ------- | ------- |
-| commits | update commits | default | not-by-default |
default | | | | |
-| commit_parents | | default | |
| | | | |
-| commit_files | | default | |
| | | | |
-| pull_requests | | | default |
default | | | | |
-| pull_request_commits | | | default |
default | | | | |
-| pull_request_comments | | | default |
default | | | | |
-| pull_request_labels | | | default |
| | | | |
-| refs | | default | |
| | | | |
-| refs_commits_diffs | | | |
| | | default | |
-| refs_issues_diffs | | | |
| | | default | |
-| ref_pr_cherry_picks | | | |
| | | default | |
-| repos | | | default |
default | | | | |
-| repo_commits | | default | default |
| | | | |
-| board_repos | | | |
| | | | |
-| issue_commits | | | |
| | | | |
-| issue_repo_commits | | | |
| | | | |
-| pull_request_issues | | | |
| | | | |
-| refs_issues_diffs | | | |
| | | | |
-| boards | | | default |
| | default | | default |
-| board_issues | | | default |
| | default | | default |
-| issue_changelogs | | | |
| | default | | default |
-| issues | | | default |
| | default | | default |
-| issue_comments | | | |
| | default | | default |
-| issue_labels | | | default |
| | | | |
-| sprints | | | |
| | default | | default |
-| issue_worklogs | | | |
| | default | | default |
-| users o | | | default |
| | default | | default |
-| builds | | | |
| default | | | |
-| jobs | | | |
| default | | | |
-
-
+This table shows the entities collected by each plugin. Domain layer entities
in this table are consistent with the entities
[here](/DataModels/DevLakeDomainLayerSchema.md).
+✅ : Collect by default.
+💪 : Collect not by default. You need to add the corresponding subtasks to
collect these entities in the [advanced
mode](../UserManuals/ConfigUI/AdvancedMode.md).
+
+| Domain Layer Entities
| ae | dora | gitextractor | incoming webhook | github | gitlab
| jenkins | jira | refdiff | tapd |
+|
-------------------------------------------------------------------------------------------
| --- | ---- | ------------ | ---------------- | ------ | ------ | ------- |
---- | ------- | ---- |
+| [accounts](../DataModels/DevLakeDomainLayerSchema.md/#accounts)
| | | | | ✅ | ✅ |
| ✅ | | ✅ |
+| [board_issues](../DataModels/DevLakeDomainLayerSchema.md/#board_issues)
| | | | | ✅ | ✅ |
| ✅ | | ✅ |
+| [board_repos](../DataModels/DevLakeDomainLayerSchema.md/#board_repos)
| | | | | ✅ | ✅ |
| ✅ | | |
+| [board_sprints](../DataModels/DevLakeDomainLayerSchema.md/#board_sprints)
| | | | | ✅ |
| | ✅ | | ✅ |
+| [boards](../DataModels/DevLakeDomainLayerSchema.md/#boards)
| | | | | ✅ | ✅ |
| ✅ | | ✅ |
+|
[cicd_pipeline_commits](../DataModels/DevLakeDomainLayerSchema.md/#cicd_pipeline_commits)
| | ✅ | | | ✅ | ✅ | ✅ |
| | |
+| [cicd_pipelines](../DataModels/DevLakeDomainLayerSchema.md/#cicd_pipelines)
| | ✅ | | | ✅ | ✅ |
✅ | | | |
+| [cicd_scopes](../DataModels/DevLakeDomainLayerSchema.md/#cicd_scopes)
| | ✅ | | | ✅ | ✅ |
✅ | | | |
+| [cicd_tasks](../DataModels/DevLakeDomainLayerSchema.md/#cicd_tasks)
| | ✅ | | 💪 | ✅ | ✅ | ✅
| | | |
+|
[commit_file_components](../DataModels/DevLakeDomainLayerSchema.md/#commit_file_components)
| | | ✅ | | | | |
| | |
+| [commit_files](../DataModels/DevLakeDomainLayerSchema.md/#commit_files)
| | | ✅ | | |
| | | | |
+|
[commit_line_change](../DataModels/DevLakeDomainLayerSchema.md/#commit_line_change)
| | | ✅ | | | |
| | | |
+| [commit_parents](../DataModels/DevLakeDomainLayerSchema.md/#commit_parents)
| | | ✅ | | |
| | | | |
+| [commits](../DataModels/DevLakeDomainLayerSchema.md/#commits)
| ✅ | | ✅ | | 💪 | 💪 |
| | | |
+| [commits_diffs](../DataModels/DevLakeDomainLayerSchema.md/#commits_diffs)
| | | | | |
| | | ✅ | |
+| [components](../DataModels/DevLakeDomainLayerSchema.md/#components)
| | | | | |
| | | | |
+|
[finished_commits_diffs](../DataModels/DevLakeDomainLayerSchema.md/#finished_commits_diffs)
| | | | | | | |
| | |
+|
[issue_changelogs](../DataModels/DevLakeDomainLayerSchema.md/#issue_changelogs)
| | | | | | |
| ✅ | | ✅ |
+|
[issue_comments](../DataModels/DevLakeDomainLayerSchema.md/#issue_commentswip)
| | | | | ✅ | |
| | | ✅ |
+| [issue_commits](../DataModels/DevLakeDomainLayerSchema.md/#issue_commits)
| | | | | |
| | ✅ | | ✅ |
+| [issue_labels](../DataModels/DevLakeDomainLayerSchema.md/#issue_labels)
| | | | | ✅ | ✅ |
| | | ✅ |
+|
[issue_repo_commits](../DataModels/DevLakeDomainLayerSchema.md/#issue_repo_commits)
| | | | | | |
| ✅ | | |
+| [issue_worklogs](../DataModels/DevLakeDomainLayerSchema.md/#issue_worklogs)
| | | | | |
| | ✅ | | ✅ |
+| [issues](../DataModels/DevLakeDomainLayerSchema.md/#issues)
| | | | | ✅ |
| | ✅ | | ✅ |
+|
[project_issue_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_issue_metrics)
| | ✅ | | | ✅ | ✅ | | ✅
| | ✅ |
+|
[project_mapping](../DataModels/DevLakeDomainLayerSchema.md/#project_mapping)
| | ✅ | | | ✅ | ✅ | ✅
| ✅ | | ✅ |
+|
[project_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_metrics)
| | ✅ | | | ✅ | ✅ | ✅
| ✅ | | ✅ |
+|
[project_pr_metrics](../DataModels/DevLakeDomainLayerSchema.md/#project_pr_metrics)
| | ✅ | | | ✅ | ✅ |
| | | ✅ |
+| [projects](../DataModels/DevLakeDomainLayerSchema.md/#project)
| | ✅ | | | ✅ | ✅ |
✅ | ✅ | | ✅ |
+|
[pull_request_comments](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_comments)
| | | | | ✅ | ✅ | |
| | |
+|
[pull_request_commits](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_commits)
| | | | | ✅ | ✅ | |
| | |
+|
[pull_request_issues](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_issues)
| | | | | ✅ | |
| | | |
+|
[pull_request_labels](../DataModels/DevLakeDomainLayerSchema.md/#pull_request_labels)
| | | | | ✅ | ✅ |
| | | |
+| [pull_requests](../DataModels/DevLakeDomainLayerSchema.md/#pull_requests)
| | | | | ✅ | ✅ |
| | | |
+| [ref_commits](../DataModels/DevLakeDomainLayerSchema.md/#ref_commits)
| | | | | |
| | | ✅ | |
+| [refs](../DataModels/DevLakeDomainLayerSchema.md/#refs)
| | | ✅ | | |
| | | ✅ | |
+|
[refs_issues_diffs](../DataModels/DevLakeDomainLayerSchema.md/#refs_issues_diffs)
| | | | | | |
| | ✅ | |
+|
[ref_pr_cherry_picks](../DataModels/DevLakeDomainLayerSchema.md/#ref_pr_cherry_picks)
| | | | | | |
| | ✅ | |
+| [repo_commits](../DataModels/DevLakeDomainLayerSchema.md/#repo_commits)
| | | ✅ | | 💪 | 💪 |
| | | |
+| [repo_snapshot](../DataModels/DevLakeDomainLayerSchema.md/#repo_snapshot)
| | | ✅ | | |
| | | | |
+| [repos](../DataModels/DevLakeDomainLayerSchema.md/#repos)
| | | | | ✅ | ✅ |
| | | |
+| [sprint_issues](../DataModels/DevLakeDomainLayerSchema.md/#sprint_issues)
| | | | | ✅ |
| | ✅ | | ✅ |
+| [sprints](../DataModels/DevLakeDomainLayerSchema.md/#sprints)
| | | | | ✅ |
| | ✅ | | ✅ |
+| [team_users](../DataModels/DevLakeDomainLayerSchema.md/#team_users)
| | | | | |
| | | | |
+| [teams](../DataModels/DevLakeDomainLayerSchema.md/#teams)
| | | | | |
| | | | |
+| [user_account](../DataModels/DevLakeDomainLayerSchema.md/#user_accounts)
| | | | | |
| | | | |
+| [users](../DataModels/DevLakeDomainLayerSchema.md/#users)
| | | | | |
| | ✅ | | ✅ |
## Data Sync Policy
@@ -72,100 +88,88 @@ This table shows the entities collected by each plugin.
Domain layer entities in
| Subtask Name | Estimated Max Number of Request | Does It
support Incremental Collection? | Does It Support Time Filter? |
| -------------------------- | ------------------------------- |
--------------------------------------- | ---------------------------- |
| CollectStatusMeta | 1 | -
| - |
-| CollectProjectsMeta | <10 | ❌
| - |
-| CollectIssueTypesMeta | <10 | ❌
| - |
-| CollectIssuesMeta | <10^4 | ✅
| ✅ |
-| CollectIssueChangelogsMeta | 1000~10^5 | ✅
| ✅ |
-| CollectAccountsMeta | <10^3 | ❌
| ❌ |
-| CollectWorklogsMeta | 1000~10^5 | ✅
| ✅ |
-| CollectRemotelinksMeta | 1000~10^5 | ✅
| ✅ |
-| CollectSprintsMeta | <100 | ❌
| ❌ |
-| CollectEpicsMeta | <100 | ❌
| ✅ |
-
-
+| CollectProjectsMeta | <10 | ❌
| - |
+| CollectIssueTypesMeta | <10 | ❌
| - |
+| CollectIssuesMeta | <10^4 | ✅
| ✅ |
+| CollectIssueChangelogsMeta | 1000~10^5 | ✅
| ✅ |
+| CollectAccountsMeta | <10^3 | ❌
| ❌ |
+| CollectWorklogsMeta | 1000~10^5 | ✅
| ✅ |
+| CollectRemotelinksMeta | 1000~10^5 | ✅
| ✅ |
+| CollectSprintsMeta | <100 | ❌
| ❌ |
+| CollectEpicsMeta | <100 | ❌
| ✅ |
### Jenkins
| Subtask Name | Estimated Max Number of Request | Does It support
Incremental Collection? | Does It Support Time Filter? |
| -------------------- | ------------------------------- |
--------------------------------------- | ---------------------------- |
-| CollectApiBuildsMeta | ≈100 | ❌
| ❌ |
-| CollectApiStagesMeta | ≈10^4 | ❌
| ✅ |
-
-
+| CollectApiBuildsMeta | ≈100 | ❌
| ❌ |
+| CollectApiStagesMeta | ≈10^4 | ❌
| ✅ |
### Gitlab
| Subtask Name | Estimated Max Number of Request | Does It
support Incremental Collection? | Does It Support Time Filter? |
| --------------------------- | ------------------------------- |
--------------------------------------- | ---------------------------- |
-| CollectApiIssuesMeta | <10^4 | ✅
| ✅ |
-| CollectApiMergeRequestsMeta | <10^3 | ✅
| ✅ |
-| CollectApiMrNotesMeta | <10^5 | ❌
| ✅ |
-| CollectApiMrCommitsMeta | <10^5 | ❌
| ✅ |
-| **CollectApiPipelinesMeta** | <10^4 | ✅
| ❌ |
-| CollectApiJobsMeta | <10^5 | ❌
| ✅ |
-
-
+| CollectApiIssuesMeta | <10^4 | ✅
| ✅ |
+| CollectApiMergeRequestsMeta | <10^3 | ✅
| ✅ |
+| CollectApiMrNotesMeta | <10^5 | ❌
| ✅ |
+| CollectApiMrCommitsMeta | <10^5 | ❌
| ✅ |
+| **CollectApiPipelinesMeta** | <10^4 | ✅
| ❌ |
+| CollectApiJobsMeta | <10^5 | ❌
| ✅ |
### Github
| Subtask Name | Estimated Max Number of Request | Does
It support Incremental Collection? | Does It Support Time Filter? |
| ---------------------------------- | ------------------------------- |
--------------------------------------- | ---------------------------- |
| --------------------------------- | Common |
----------------------- | |
-| CollectMilestonesMeta | ≈10 | ✅
| ❌ |
-| CollectRunsMeta | <10^4 | ❌
| ✅ |
-| CollectApiCommentsMeta | 400 (max page github supported) | ✅
| ❌ |
-| **CollectApiEventsMeta** | 400 (max page github supported) | ❌
| ❌ |
-| CollectApiPullRequestReviewsMeta | <10^5 | ✅
| ✅ |
+| CollectMilestonesMeta | ≈10 | ✅
| ❌ |
+| CollectRunsMeta | <10^4 | ❌
| ✅ |
+| CollectApiCommentsMeta | 400 (max page github supported) | ✅
| ❌ |
+| **CollectApiEventsMeta** | 400 (max page github supported) | ❌
| ❌ |
+| CollectApiPullRequestReviewsMeta | <10^5 | ✅
| ✅ |
| --------------------------------- | Graphql Only (Default) |
----------------------- | |
-| CollectIssueMeta | ≈10^4 | ❌
| ✅ |
-| CollectPrMeta | ≈10^3 | ❌
| ✅ |
-| CollectCheckRunMeta | <10^4 | ❌
| ✅ |
-| CollectAccountMeta | ≈10^2 | ❌
| - |
+| CollectIssueMeta | ≈10^4 | ❌
| ✅ |
+| CollectPrMeta | ≈10^3 | ❌
| ✅ |
+| CollectCheckRunMeta | <10^4 | ❌
| ✅ |
+| CollectAccountMeta | ≈10^2 | ❌
| - |
| --------------------------------- | Restful Only (Not Default) |
----------------------- | |
-| CollectApiIssuesMeta | ≈10^4 | ✅
| ❌ |
-| CollectApiPullRequestsMeta | ≈10^2 | ✅
| ❌ |
-| CollectApiPullRequestCommitsMeta | ≈10^4 | ✅
| ✅ |
-| **CollectApiPrReviewCommentsMeta** | ≈10^4 | ✅
| ❌ |
-| **CollectAccountsMeta** | ≈10^4 | ❌
| ❌ |
-| **CollectAccountOrgMeta** | ≈10^4 | ❌
| ❌ |
-| CollectJobsMeta | <10^6 | ❌
| ✅ |
+| CollectApiIssuesMeta | ≈10^4 | ✅
| ❌ |
+| CollectApiPullRequestsMeta | ≈10^2 | ✅
| ❌ |
+| CollectApiPullRequestCommitsMeta | ≈10^4 | ✅
| ✅ |
+| **CollectApiPrReviewCommentsMeta** | ≈10^4 | ✅
| ❌ |
+| **CollectAccountsMeta** | ≈10^4 | ❌
| ❌ |
+| **CollectAccountOrgMeta** | ≈10^4 | ❌
| ❌ |
+| CollectJobsMeta | <10^6 | ❌
| ✅ |
| CollectApiCommitsMeta | Not enabled | -
| - |
| CollectApiCommitStatsMeta | Not enabled | -
| - |
-
-
### Feishu
| Subtask Name | Estimated Max Number of Request | Does It
support Incremental Collection? | Does It Support Time Filter? |
| ----------------------------- | ------------------------------- |
--------------------------------------- | ---------------------------- |
-| CollectMeetingTopUserItemMeta | ≈10^3 | ❌
| ✅ |
-
-
+| CollectMeetingTopUserItemMeta | ≈10^3 | ❌
| ✅ |
### Bitbucket
| Subtask Name | Estimated Max Number of Request | Does
It support Incremental Collection? | Does It Support Time Filter? |
| ----------------------------------- | ------------------------------- |
--------------------------------------- | ---------------------------- |
-| ~~CollectApiRepoMeta~~ | 1 | ❌
| ❌ |
-| CollectApiPullRequestsMeta | ≈10^3 | ❌
| ❌ |
-| **CollectApiIssuesMeta** | ≈10^4 | ❌
| ❌ |
-| **CollectApiPrCommentsMeta** | ≈10^5 | ❌
| ❌ |
-| **\*CollectApiIssueCommentsMeta\*** | ≈10^6 | ❌
| ❌ |
-| **CollectApiPipelinesMeta** | <10^4 | ❌
| ❌ |
-| CollectApiDeploymentsMeta | <10^2 | ❌
| ❌ |
-
-
+| ~~CollectApiRepoMeta~~ | 1 | ❌
| ❌ |
+| CollectApiPullRequestsMeta | ≈10^3 | ❌
| ❌ |
+| **CollectApiIssuesMeta** | ≈10^4 | ❌
| ❌ |
+| **CollectApiPrCommentsMeta** | ≈10^5 | ❌
| ❌ |
+| **\*CollectApiIssueCommentsMeta\*** | ≈10^6 | ❌
| ❌ |
+| **CollectApiPipelinesMeta** | <10^4 | ❌
| ❌ |
+| CollectApiDeploymentsMeta | <10^2 | ❌
| ❌ |
### Gitee
| Subtask Name | Estimated Max Number of Request |
Does It support Incremental Collection? | Does It Support Time Filter? |
| ------------------------------------ | ------------------------------- |
--------------------------------------- | ---------------------------- |
-| ~~CollectApiRepoMeta~~ | 1 | ❌
| ❌ |
-| CollectApiPullRequestsMeta | ≈10^3 | ✅
| ❌ |
-| **CollectApiIssuesMeta** | ≈10^4 | ✅
| ❌ |
-| **CollectCommitsMeta?** | ≈10^4 | ✅
| ❌ |
-| **CollectApiPrCommentsMeta** | ≈10^5 | ❌
| ❌ |
-| **\*CollectApiIssueCommentsMeta\*** | ≈10^6 | ✅
| ❌ |
-| **CollectApiPullRequestCommitsMeta** | ≈10^5 | ❌
| ❌ |
-| **CollectApiPullRequestReviewsMeta** | ≈10^5 | ❌
| ❌ |
-| **\*CollectApiCommitStatsMeta\*** | ≈10^6 (Not enable) | ❌
| ❌ |
+| ~~CollectApiRepoMeta~~ | 1 | ❌
| ❌ |
+| CollectApiPullRequestsMeta | ≈10^3 | ✅
| ❌ |
+| **CollectApiIssuesMeta** | ≈10^4 | ✅
| ❌ |
+| **CollectCommitsMeta?** | ≈10^4 | ✅
| ❌ |
+| **CollectApiPrCommentsMeta** | ≈10^5 | ❌
| ❌ |
+| **\*CollectApiIssueCommentsMeta\*** | ≈10^6 | ✅
| ❌ |
+| **CollectApiPullRequestCommitsMeta** | ≈10^5 | ❌
| ❌ |
+| **CollectApiPullRequestReviewsMeta** | ≈10^5 | ❌
| ❌ |
+| **\*CollectApiCommitStatsMeta\*** | ≈10^6 (Not enable) | ❌
| ❌ |
