Trevor Edwards created AIRFLOW-1504:

             Summary: Log Cluster Name on Dataproc Operator When Executed
                 Key: AIRFLOW-1504
             Project: Apache Airflow
          Issue Type: Improvement
            Reporter: Trevor Edwards
            Priority: Minor

Since Jinja template substitution happens only just before execution of a task 
instance, it is not possible to inspect the cluster name populated for a 
Dataproc operator. This makes it hard to debug issues such as using ts_nodash 
as part of the clustername, as printing when the DAG is read would give the 

Cluster name: smoke-cluster-{{ ts_nodash }}

then the operator would execute and the user receives the error:

<HttpError 400 when requesting ... returned "clusterName must be a match of 
regex '(?:[a-z](?:[-a-z0-9]{0,49}[a-z0-9])?).'">

but the user has no idea what cluster name was provided to the API.

See attached example pipeline.

This message was sent by Atlassian JIRA

Reply via email to