eye-gu opened a new issue, #18335: URL: https://github.com/apache/dolphinscheduler/issues/18335
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement. ### Description ### Problem Statement In DolphinScheduler, task resource configurations (e.g., Flink slots/parallelism, Shell CPU quotas, worker groups) are **statically defined** at task definition time via the UI and **cannot be dynamically adjusted at runtime**. This creates two key pain points: 1. **Data-driven resource allocation**: When upstream data volume fluctuates significantly, downstream tasks cannot dynamically scale their resource allocation based on upstream computation results. 2. **API-driven scheduling**: When triggering workflows via API (e.g., from an external orchestration system), there is no way to adjust task resource configs based on current cluster load. ### Solution Reuse the existing parameter passing mechanism by introducing a dsInternal.* namespace as DolphinScheduler's internal parameter convention. This allows upstream tasks or API callers to dynamically override downstream task configurations without modifying task definitions. ### Core Mechanism 1. Namespace: dsInternal.<taskDefinitionName>.<fieldName> — uses dsInternal. as a fixed prefix to avoid conflicts with user business parameters. 2. Zero-config passthrough: dsInternal.* parameters do not require IN/OUT declarations in custom parameter configuration. The system automatically injects and passes them unconditionally. 3. Override sources: - Upstream task: Output via Shell #{setValue(dsInternal.flink_etl_task.slot=4)} - API trigger: Pass in startParams as "dsInternal.flink_etl_task.slot": "4" 4. UI hint: Display the corresponding dsInternal parameter name (e.g., dsInternal.flink_etl_task.slot) next to each resource config field in the task editor, with a copy button for convenience. 5. Fully backward compatible — behavior is unchanged when no dsInternal.* parameters are present. ### Use case Workflow DAG: [data_size_check: Shell] → [flink_etl_task: Flink] # Upstream Shell dynamically adjusts Flink resources based on data size DATA_SIZE=$(hdfs dfs -du -s /data/input | awk '{print $1}') if [ $DATA_SIZE -gt 10737418240 ]; then echo '#{setValue(dsInternal.flink_etl_task.slot=4)}' echo '#{setValue(dsInternal.flink_etl_task.parallelism=16)}' else echo '#{setValue(dsInternal.flink_etl_task.slot=2)}' echo '#{setValue(dsInternal.flink_etl_task.parallelism=4)}' fi # Downstream flink_etl_task was created with slot=2, now overridden to slot=4 ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
