GitHub user Hui-of-limin edited a discussion: Dolphin MCP Pilot: Operate DolphinScheduler via AI Agents
# Dolphin MCP Pilot: Operate DolphinScheduler via AI Agents Hi DolphinScheduler community ๐ We've built **Dolphin MCP Pilot**, an MCP (Model Context Protocol) server that lets AI agents directly operate DolphinScheduler through natural language โ from creating workflows and managing schedules to troubleshooting failures and performing backfills. ## What is it? An integration layer that exposes **58 DolphinScheduler operations** as standardized MCP tools, enabling AI agents (Claude Desktop, Cursor, Cline, custom agents) to: - Create SQL/DAG workflows with a single sentence - Manage schedule lifecycle (create โ online โ offline โ delete) - Control process instances (pause / resume / rerun / rerun-from-failure) - Intervene at task level (force success / skip failed nodes) - Query logs and troubleshoot failures with guided `next_action` prompts - Perform serial backfills with ordering guarantees (`complementStartDate`/`complementEndDate`) - Manage resources and roll back workflow versions ## Why this matters Traditional DolphinScheduler interaction requires: 1. Opening the web UI 2. Navigating multiple pages 3. Manually clicking through operations 4. Checking logs and states separately With Dolphin MCP Pilot, you can tell your AI agent: > "The ETL job failed yesterday. Why?" The agent automatically: 1. Calls `ds_list_process_instances` โ finds the failed instance 2. Calls `ds_list_task_instances` โ locates the failed task node 3. Calls `ds_get_task_log` โ pulls the error log 4. Reports the root cause No manual clicking, no page-hopping. ## Architecture - **Protocol**: MCP over HTTP + SSE - **Authentication**: Dual-mode (API Token or Username+Password) - **Compatibility**: DolphinScheduler 3.x (tested on 3.2.1) - **Deployment**: Docker / bare Python ## Tool Coverage | Category | Tools | Examples | |---|---|---| | Project | 5 | `ds_create_project`, `ds_list_projects` | | Workflow | 14 | `ds_create_workflow`, `ds_update_task_param`, `ds_clone_workflow` | | Schedule | 6 | `ds_set_schedule`, `ds_online_schedule`, `ds_complement_data` | | Instance | 13 | `ds_rerun_process_instance`, `ds_force_task_success` | | Resource | 10 | `ds_list_resources`, `ds_update_resource_content` | | Monitoring | 6 | `ds_get_task_log`, `ds_list_task_instances` | | Raw API | 4 | `ds_raw_get`, `ds_raw_post`, `ds_raw_put`, `ds_raw_delete` | ## Quick Start ```bash git clone https://github.com/iflytek/dolphin-mcp-pilot.git cd dolphin-mcp-pilot # Configure cp .env.example .env # Edit .env: set DS_URL, DS_TOKEN (or DS_USER/DS_PASSWORD) # Run docker compose --profile dev up -d ``` Connect your AI agent (example: Claude Desktop): ```json { "mcpServers": { "dolphinscheduler": { "url": "http://localhost:8001/mcp/";, "headers": { "X-DS-Token": "your-token-here" } } } } ``` Now you can talk to your agent: > "Create a daily SQL workflow in the data-team project that runs at 2 AM, > querying `SELECT * FROM user_behavior WHERE dt = '${bizdate}'`, with 3 > retries on failure." The agent orchestrates `ds_create_workflow` โ `ds_set_schedule` โ `ds_online_schedule` automatically. ## Real-World Use Cases **Scenario 1**: Automated troubleshooting - Agent detects a failed workflow, traces the specific task node, pulls logs, and suggests fixes **Scenario 2**: Bulk backfills - "Rerun all daily reports from July 1 to July 31" โ agent calls `ds_complement_data` with serial ordering **Scenario 3**: Version control - "Roll back user_analysis workflow to the previous version" โ agent calls `ds_rollback_workflow_version` ## Roadmap - DolphinScheduler 2.x compatibility - Pre-built DAG templates (CDC, feature engineering, report distribution) - Multi-agent orchestration (scheduling agent โ data quality agent โ alerting agent) ## Links - **GitHub**: https://github.com/iflytek/dolphin-mcp-pilot - **License**: Apache-2.0 - **Docs**: [Installation](https://github.com/iflytek/dolphin-mcp-pilot/blob/main/docs/INSTALLATION.md) ยท [Configuration](https://github.com/iflytek/dolphin-mcp-pilot/blob/main/docs/CONFIGURATION.md) ยท [Features](https://github.com/iflytek/dolphin-mcp-pilot/blob/main/docs/FEATURES.md) ยท [API Reference](https://github.com/iflytek/dolphin-mcp-pilot/blob/main/docs/API.md) --- We'd love to hear feedback from the community, especially on: - API compatibility issues with different DS versions - Authentication patterns in production environments - Feature requests for operations not yet covered This is an open-source project under Apache-2.0. Contributions, issues, and discussions are welcome! GitHub link: https://github.com/apache/dolphinscheduler/discussions/18547 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
