cshaddox opened a new issue, #51598: URL: https://github.com/apache/airflow/issues/51598
### Apache Airflow Provider(s) databricks ### Versions of Apache Airflow Providers 6.12.0 ### Apache Airflow version 2.10.3 ### Operating System managed airflow ### Deployment Docker-Compose ### Deployment details _No response_ ### What happened When using DatabricksWorkflowTaskGroups, a launch task is created. This launch task can be set to have upstream dependencies, however no matter what the launch task will not wait on any dependencies to execute, executing as soon as the dag run starts. ### What you think should happen instead This task should instead wait on prior tasks to finish as aligned with the trigger_rule assigned to that specific task. As the trigger_rule by default for the launch task is set to all_success, the task should NOT run until all upstream dependencies have finished. <img width="674" alt="Image" src="https://github.com/user-attachments/assets/83abb3ce-1d36-4e12-b485-b0855a0caf2b"; /> <img width="982" alt="Image" src="https://github.com/user-attachments/assets/836fcfe2-21c4-4784-b470-ac2d2dca2e54"; /> ### How to reproduce ```from airflow.models import DAG from airflow.operators.dummy import DummyOperator from airflow.providers.databricks.operators.databricks_workflow import ( DatabricksWorkflowTaskGroup, ) with DAG( dag_id="example_dag", schedule_interval=f"0 0 * * *", ) as dag: start_task = DummyOperator(task_id="end", trigger_rule="none_failed") task_group = DatabricksWorkflowTaskGroup( group_id="tg", databricks_conn_id="FILL IN DBX CONNECTION ID", job_clusters={ "job_cluster_key": "test_cluster", "new_cluster": { "cluster_name": "", "spark_version": "15.4.x-scala2.12", "node_type_id": "r4.2xlarge", "num_workers": 1, "autoscale": {"min_workers": 2, "max_workers": 8}, "aws_attributes": { "availability": "SPOT_WITH_FALLBACK", "ebs_volume_count": 1, "ebs_volume_size": 100, "ebs_volume_type": "GENERAL_PURPOSE_SSD", "first_on_demand": 1, "spot_bid_price_percent": 100, "zone_id": "auto", }, }, }, ) with task_group: pass ### Anything else _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
