jaketf opened a new issue #8672: URL: https://github.com/apache/airflow/issues/8672
[Here](https://github.com/apache/airflow/blob/4421f011eeec2d1022a39933e27f530fb9f9c1b1/airflow/providers/google/cloud/hooks/datafusion.py#L392) the `start_pipeline` method of data fusion hook will succeed if they get a 200 from the CDAP API. This is a misleading success signal as this indicates at best that this pipeline entered the PENDING state. However start pipeline should not succeed until the pipline has reached the RUNNING state. Note the Happy path is PENDING > STARTING> RUNNING ([ProgramStatus](https://github.com/cdapio/cdap/blob/1d62163faaecb5b888f4bccd0fcf4a8d27bbd549/cdap-proto/src/main/java/io/cdap/cdap/proto/ProgramRunStatus.java)) Many CDAP pipelines using Dataproc Provisioner spend a significant amount of time in the STARTING state because they also have tick through the various [ProgramRunClusterStatus](https://github.com/cdapio/cdap/blob/1d62163faaecb5b888f4bccd0fcf4a8d27bbd549/cdap-proto/src/main/java/io/cdap/cdap/proto/ProgramRunClusterStatus.java) for provisioning the dataproc cluster. Unfortunately making the start call to CDAP does not return a run_id. This hook could work around this by adding a special runtime arg called `__faux_airflow_id__` which can be used to "look up" the real run id by this special runtime arg. the value of this runtime arg could be the dag_run_id or something. If using this workaround or CDAP API can return run id, then a more useful operator than start pipeline would be one that actually waits til the job reaches a success state (much like the existing dataflow and dataproc operators). Example in golang for terraform provider resource that manages a streaming pipeline. [Creating with faux id](https://github.com/GoogleCloudPlatform/terraform-provider-cdap/blob/master/cdap/resource_streaming_program_run.go#L108) And [looking up the real CDAP run id by faux id](https://github.com/GoogleCloudPlatform/terraform-provider-cdap/blob/master/cdap/resource_streaming_program_run.go#L216) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
