eye-gu commented on issue #18350:
URL:
https://github.com/apache/dolphinscheduler/issues/18350#issuecomment-4750899752
### Git local cache
```mermaid
flowchart TB
subgraph DS["DolphinScheduler"]
subgraph API["API Server (cache disabled)"]
API_OP["GitStorageOperator<br/>upload / download / delete / copy
/ list"]
API_CC["GitApiClient<br/>(REST API)"]
API_OP -->|"A1"| API_CC
end
subgraph WK["Worker(s)"]
TASK["Task"]
WK_OP["GitStorageOperator<br/>cache.enabled?<br/>(all reads:
download/fetch/list/exists/get)"]
CACHE["GitLocalCacheManager<br/>R3: checkFreshness()<br/>R4:
read local clone"]
LOCAL["Local Git Clone<br/>(disk: cacheDir/)"]
BG["Background Refresh<br/>B1: ls-remote → fetch --depth 1<br/>+
reset --hard (if SHA changed)<br/>every refreshIntervalSec"]
TASK -->|"R1 read"| WK_OP
WK_OP -->|"R2 cache enabled"| CACHE
WK_OP -.->|"cache disabled → REST API<br/>"|
API_CC2["GitApiClient<br/>(REST API)"]
CACHE --> LOCAL
BG -->|"reset --hard"| LOCAL
end
end
REMOTE["Git Remote Repository<br/>(GitHub / Gitea / GitLab)<br/>Single
Source of Truth"]
API_CC -->|"A2 Git REST API<br/>GET / PUT /
DELETE<br/>/repos/{owner}/{repo}/contents/{path}"| REMOTE
BG -->|"B1 git fetch"| REMOTE
BG -.->|"B0 git clone (initial)"| REMOTE
API_CC2 -.->|"REST API"| REMOTE
```
**Three paths:**
**Path A — API Server（cache disabled, read/write through by REST API）:**
```
A1 GitStorageOperator(upload/download/delete/copy/list) → GitApiClient
A2 GitApiClient → Git Remote, via Git Rest API:
GET / PUT / DELETE /repos/{owner}/{repo}/contents/{path}?ref={branch}
```
**Path R — Worker （cache enabled）:**
```
R1 Task read
request(download/fetchFileContent/listStorageEntity/exists/getStorageEntity)
→ GitStorageOperator(cache.enabled?)
R2 cache enabled → GitLocalCacheManager
R3 checkFreshness(): now - lastRefreshTime > maxStaleness?
├── YES → throw exception (fail-fast, reject stale data)
└── NO → continue
R4 read from Local Git Clone → return content (zero network cost)
─────────────────────────────────────────────────────────────
cache disabled → GitApiClient(REST API) → Git Remote
```
**Path B — Worker （Background Refresh）:**
```
B0 initial: git clone --depth 1 (Worker startup and local cache not exist)
B1 every refreshIntervalSec:
ls-remote → SHA equal? skip : fetch --depth 1 + reset --hard
success → lastRefreshTime = now
failure → no update, timestamp ages, eventually triggers R3 stale guard
```
Core idea: **API Server reads and writes all go through REST API (ensuring
write-then-immediately-visible); Worker has only read operations, served from
local cache (background periodic refresh + maxStaleness expiry protection)**.
- **API Server cache disabled**: `upload`, `download`, `delete`, `copy`,
`listStorageEntity`, etc. all go through REST API, bypassing the cache
- **Worker cache enabled**: read-only; all read operations go through the
local clone, zero network overhead
- **Refresh** (Worker): a background periodic task runs `git fetch` + `git
reset --hard`, the sole refresh path
### Design Decisions
| Decision | Choice | Rationale |
| --- | --- | --- |
| API Server cache disabled | Always REST API | Ensures user writes are
immediately visible, avoids cache-induced user confusion |
| Worker cache enabled | Local cache | High-frequency reads during task
execution; local I/O eliminates network overhead |
| Read-only mode | `git.read-only` switch, default off | Remote repo may be
managed by CI, DS is read-only; the configured token may lack write permission,
so writes must be rejected fast at the entry point |
| Cache granularity | Entire repo local clone | Git manages at the repo
unit; avoids fragmentation and consistency issues of per-file caching |
| Refresh strategy | Background periodic refresh (sole path) | Read path
triggers no network operations, deterministic performance; refresh fully
decoupled from read |
| Staleness protection | `maxStaleness` threshold check | When background
refresh fails consecutively beyond the threshold, reject cache reads and error
out, avoiding use of stale files (likely the remote repo has issues; falling
back to REST API is meaningless) |
| Git operation library | JGit | Pure Java, no need to install git CLI on
Worker |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
