dijiekstra commented on issue #18627:
URL:
https://github.com/apache/dolphinscheduler/issues/18627#issuecomment-5615293526
## 📅 Suggested Implementation Timeline
### Phase 1 - MVP (2–3 weeks)
**Goal**: complete the minimum closed loop for Paimon single-asset
snapshot-driven triggering
**Week 1**
- Day 1–2: database table design and migration scripts
- Day 3–5: core `DependencyResolverService` implementation (snapshot-only
evaluation)
- Day 5–10: `ASSET_SENSOR` task plugin + `AssetSensorTracker`
**Week 2**
- Day 1–3: implement Paimon `AssetEventScanner`
- Day 4–5: Master integration (`TaskProcessor` checks + configuration)
- Day 5–10: unit tests and integration tests
**Week 3**
- Day 1–3: end-to-end testing (GMV scenario)
- Day 4–5: performance tuning and bug fixes
- Day 6–10: documentation and example preparation
### Phase 2 - Expansion (3–4 weeks)
**Goal**: support multi-asset AND/OR, Watermark/Quality/Schema conditions,
and compensation mechanisms
**Week 1**
- Multi-asset AND/OR dependencies (`DependencyResolver` improvements)
- Watermark/Quality/Schema condition evaluation
- Workflow-level `ASSET_EVENT_TRIGGER` design
**Week 2**
- `CompensationScanner` logic
- Push event receiving endpoint
- Related DAO and Service implementations
**Week 3**
- Testing (especially out-of-order and high-concurrency scenarios)
- Documentation updates
**Week 4**
- Performance testing and tuning
- Community feedback handling
### Phase 3 - Ecosystem (as needed)
**Goal**: complete Lakehouse support and user experience
- Iceberg/Hudi event sources
- UI asset dashboard and dependency visualization
- Alerting rules and observability
- Performance optimization (cache, batching, etc.)
---
## 👥 Suggested team breakdown
### Database and DAO layer (1 person, 0.5 week)
- DDL design for 6 tables
- Migration scripts
- DAO interfaces and implementations
### Core Service layer (2 people, 2 weeks)
- `DependencyResolverService`
- `AssetStateService` / `AssetEventService`
- DependencyResolver business logic
### Task plugin (1 person, 1 week)
- `ASSET_SENSOR` task type
- `AssetSensorTracker` implementation
- Integration with Master
### Event source (1 person, 1 week)
- `AssetEventScanner` (Paimon polling)
- Event processing pipeline
- Resume-from-offset scanning
### Master integration (1 person, 0.5 week)
- `TaskProcessor` changes
- `CommandType` extension (optional)
- Configuration system
### Testing and documentation (1 person, 1.5 weeks)
- Unit test coverage
- Integration test scripts
- User documentation and examples
---
## 💼 Estimated workload
| Module | Workload | Complexity | Dependencies |
|-----|--------|--------|------|
| Database | 0.5 week | Low | None |
| DependencyResolverService | 1.5 weeks | Medium | Database |
| ASSET_SENSOR plugin | 1 week | Medium | DependencyResolver |
| AssetEventScanner | 1 week | Medium | Database |
| Master integration | 0.5 week | Low | ASSET_SENSOR |
| Testing & Documentation | 1.5 weeks | Low | All modules |
| **Total** | **5.5 weeks** | - | - |
**Estimated for a 3-person team**: 4–5 weeks to complete the MVP
**Estimated for a 2-person team**: 6–7 weeks to complete the MVP
---
## 🎯 MVP success criteria
- ✅ 6 tables are designed correctly and migration scripts are executable
- ✅ `DependencyResolver` can correctly evaluate single-asset snapshot
dependencies
- ✅ `ASSET_SENSOR` tasks can be created, executed, and released
- ✅ Paimon snapshot advancement can be detected and events can be collected
- ✅ End-to-end flow works completely: snapshot advancement → task release →
DAG execution
- ✅ Idempotency is verified under multi-Master concurrency
- ✅ Complete unit test coverage exists (idempotency, deduplication,
out-of-order handling, AND/OR)
- ✅ Documentation is clear and examples are runnable
---
## 🔗 Expected PR breakdown
Expected PRs for the MVP phase:
1. **Infrastructure** (database)
- PR: Add `t_ds_asset*` tables and migration scripts
2. **Core Service** (core logic)
- PR: Implement `DependencyResolverService`
- PR: Implement `AssetStateService` and related DAOs
3. **Task Plugin** (task type)
- PR: Add `ASSET_SENSOR` task plugin
4. **Event Source** (event collection)
- PR: Implement `AssetEventScanner` for Paimon
5. **Master Integration** (scheduler engine changes)
- PR: Integrate asset-event triggering into Master
6. **Testing & Docs** (tests and documentation)
- PR: Add unit/integration tests for asset-event scheduling
- PR: Add a user guide and examples
---
## âš¡ Suggested quick-start actions
**If work starts immediately, the recommended first steps are:**
1. Confirm with the community the applicability scope of the paradigm shift
(pure snapshot model)
2. Confirm whether the MVP workload estimate is acceptable
3. Start parallel work on:
- Final confirmation of database design
- Discussion of the `DependencyResolver` interface contract
- API design for the `ASSET_SENSOR` task type
4. Complete within the first week:
- Database DDL and migration scripts
- `DependencyResolver` interface and unit test framework
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
