github-actions[bot] commented on issue #18625:
URL:
https://github.com/apache/dolphinscheduler/issues/18625#issuecomment-5615519823
# RFC: Lake warehouse asset version event-driven scheduling design
## Overview
This RFC proposes to introduce **asset event-driven scheduling
capabilities** to Apache DolphinScheduler to support data version-driven
workflow scheduling under the modern CDC lake warehouse architecture
(Paimon/Iceberg/Hudi).
## Core Objectives
Upgrade DolphinScheduler from "time DAG scheduling system" to "data asset
version driven scheduling system" so that:
```text
Lake warehouse table snapshot promotion
→ The event is sensed and persisted as an AssetEvent
→ Update the current state of the asset AssetState
→ Match evaluation based on dependency conditions
→ Trigger/release DolphinScheduler workflow or task instance
→ Execution, observation, failure retry and compensation
```
## Design Summary
### Applicable scenarios
- ✅ **New generation data warehouse architecture**: ODS→DWD→DWS→ADS full
link based on Paimon/Iceberg/Hudi
- ✅ **Pure snapshot driver**: The entire DAG is completely determined by the
upstream asset snapshot version advancement, no time or cron is involved.
- ✅ **Multi-Asset Coordination**: Middle layers such as DWD rely on multiple
ODS asset snapshots to be ready at the same time
- ❌ **Not applicable**: Data warehouses still using traditional time
triggering
### Core Design Decisions
1. **Do not change the Master kernel** - Reuse the existing
CommandType/Command/WorkflowInstance/TaskInstance state machine
2. **Plug-in integration** - Added ASSET_SENSOR task type,
DependencyResolverService as the Service layer
3. **Impotent first** - trigger_key + database unique constraint guarantees
single triggering
4. **Reliable Compensation** - Push event + Poll compensation +
CompensationScanner takes care of everything
5. **Phased implementation** - MVP (Paimon single asset) → multi-asset
AND/OR → Iceberg/Hudi
## Key architectural elements
### Add new database table (6 photos)
```sql
t_ds_asset -- asset metadata
t_ds_asset_event -- original event (evidence)
t_ds_asset_state -- current state of the asset (source of dispatch truth)
t_ds_asset_dependency -- workflow/task dependency statement on assets
t_ds_asset_trigger_history -- idempotent trigger history
t_ds_asset_event_consumer_offset -- event consumption location
```
### Three-layer dependency determination
| Level | Judgment method | Timing |
|-----|--------|------|
| Workflow level | DependencyResolver evaluates t_ds_asset_dependency |
Before creating WorkflowInstance (Strategy C, Phase 2) |
| Task level | DAG front and rear dependencies + ASSET_SENSOR polling +
DEPENDENT | TaskProcessor before submitting to Worker |
| DAG internal | Normal topology execution | Automatically after task
completion |
### Master transformation (< 500 lines of code)
5 modification points, all plug-in:
1. **DependencyResolver integration** - New Service evaluates asset
dependencies
2. **ASSET_SENSOR Task** - New task type + AssetSensorTracker
3. **Workflow startup check** - Verification after WorkflowInstance creation
(optional)
4. **Compensation Scanner** - AssetEventCompensationScanner (5~15 minute
cycle)
5. **Configuration item** - dolphinscheduler.asset-event-scheduling.*
## Implementation Roadmap
### Phase 1 (MVP, 2~3 weeks)
- [ ] 6 tables DDL + Migration script
- [ ] DependencyResolverService (only snapshot determination)
- [ ] ASSET_SENSOR task plug-in + AssetSensorTracker
- [ ] Paimon AssetEventScanner (polling mode)
- [ ] End-to-end testing: snapshot advancement → task release → DAG execution
### Phase 2 (Extended, 3~4 weeks)
- [ ] Multi-asset AND/OR dependencies
- [ ] Watermark/Quality/Schema conditions
- [ ] CompensationScanner compensation logic
- [ ] Workflow-level ASSET_EVENT_TRIGGER
### Phase 3 (ecological)
- [ ] Iceberg/Hudi event source
- [ ] Push event receiving HTTP endpoint
- [ ] UI: Asset Market + Dependency Visualization
- [ ] Alert rules and observability dashboard
## End-to-end example: GMV scenario
```text
MySQL CDC → Paimon CDC write
↓
ODS_order_latest, ODS_payment_latest snapshot advancement
↓
DWD_trade_event_fact waits for both ODS snapshots to be READY (AND dependent)
↓
DWD_trade_event_fact executes and produces snapshot snapshot_id=2001
↓
DWS_gmv_daily waits for DWD_trade_event_fact snapshot to advance
↓
DWS_gmv_daily executes and produces snapshot snapshot_id=3001
↓
ADS_gmv_dashboard waiting for DWS_gmv_daily snapshot to advance
↓
The entire DAG is entirely driven by snapshot version advancement, no cron
or time expressions required
```
## Concurrency and security
### Multiple Master scenarios
- **Command creation to remove duplicates**: depends on PK or sequence
- **trigger_history unique constraint**: uk_trigger_key guarantees a single
creation
- **No need for distributed locks**: database constraints naturally handle
race conditions
### Idempotence
```
trigger_key = workflow_code + dependency_group + snapshot_id_combo
The same trigger_key can successfully create a trigger_history record at
most once.
=> The same asset snapshot combination triggers the workflow at most once
=> Natural guarantee of idempotence
```
## Observational Design
### 22+ Prometheus metrics
Covers: event ingestion, status update, dependency analysis, triggering, and
compensation.
### 20+ structured log fields
eventId, assetKey, snapshotId, triggerKey, workflowCode, taskCode, status,
reason, etc.
### Alarm rules
8 alarms: status lag, waterfall failure, compensation backlog, trigger
failure, etc.
## Impact on existing architecture
### ✅ No changes (zero risk)
- WorkflowInstance/TaskInstance core state machine
- DAG pre- and post-dependency expression and execution
- Existing logic of DEPENDENT task
- Worker task execution life cycle
- Alert/Monitor/Backfill and other peripheral functions
### ⚠️ Requires modification (plug-in, low risk)
- Added ASSET_SENSOR task type (like DependentTask)
- Added DependencyResolverService (Service layer, no kernel changes)
- Add 1 line of dependency check before TaskProcessor#submitTask()
- Added CommandType.ASSET_EVENT_TRIGGER (optional)
### Scale of code changes
**Estimated < 500 lines** (compared to tens of thousands of lines of code in
DolphinScheduler)
## Risks and Open Issues
### 8 identified risks
1. Definition of backfill and event-driven semantics
2. Race window triggered by multiple Masters concurrently
3. Task polling frequency and database pressure
4. Compatibility of CommandType semantic extension
5. Permission model for cross-catalog/cross-project asset dependencies
6. Clock/sequence dependency of Push/Poll dual path deduplication
7. exactly-once declaration scope
8. Mixed dependencies between partitioned and non-partitioned tables
All risks have been analyzed and most can be avoided through design
specifications.
## Design documentation
**Full RFC document** (line 1178) has been prepared at:
- `/docs/docs/en/architecture-design/lakehouse-asset-event-scheduling.md`
Contains:
- ✅ Background and target (paradigm shift instructions)
- ✅ Existing infrastructure assessment (Command/DEPENDENT, etc.)
- ✅ Overall architecture (domain model, event flow, database table, trigger
link)
- ✅ Integrated solutions (Strategy A/C, API, UI)
- ✅ **Detailed explanation of Master transformation** (Chapter 4.5, new line
346)
- ✅ End-to-end example (complete interpretation of GMV scenario)
- ✅ Observational design (indicators, logs, alarms)
- ✅ Asset identification specifications (AssetKey naming, registration
discovery)
- ✅ Implementation Roadmap (Phase 1/2/3)
- ✅ Architecture upgrade guide
- ✅ FAQ (7 Frequently Asked Questions)
- ✅Risk analysis (8 items)
## Discussion and Feedback
Welcome feedback and discussion from the community:
- **Functional Completeness**: Are necessary metadata, event types, or
dependencies missing?
- **Master Transformation**: Are the code locations and change scale of the
5 transformation points reasonable?
- **Performance Impact**: Are CompensationScanner's cycle and BatchSize
reasonable?
- **Permission Model**: How to standardize the design of permissions that
depend on cross-project assets?
- **Compensation Strategy**: Is the historical snapshot triggered in the
complement scenario?
- **Ecological Support**: Is Iceberg/Hudi’s event model suitable?
## Next step
1. Community review of this RFC
2. Confirm the applicable scope and limitations of paradigm shift
3. Discuss the workload distribution of MVP
4. Start Phase 1 development (if approved)
---
**Related tags**: design, enhancement, lakehouse, event-driven, scheduling,
paimon, iceberg, hudi
**Priority**: Medium (new feature, does not affect existing features)
**Type**: RFC/Discussion/Design
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
