[jira] (AIRFLOW-820) Standardize GCP related connection id names and default values

[Feng Lu (JIRA)]

Title: Message Title

Feng Lu created an issue   Apache Airflow / AIRFLOW-820 Standardize GCP related connection id names and default values Issue Type: Improvement Assignee: Feng Lu Components: contrib Created: 30/Jan/17 23:12 Priority: Major Reporter: Feng Lu A number of Google Cloud Platform (GCP) related operators, such as BigQueryCheckOperator or DataFlowJavaOperator, are using different connection_id var names and default values. For example, BigQueryCheckOperator(.., big_query_conn_id='bigquery_default'..) DataFlowJavaOperator(..., gcp_conn_id='google_cloud_default'...) DataProcClusterCreateOperator(..., google_cloud_conn_id='google_cloud_default',...) This makes dag-level default_args problematic, one would have to specify each connection_id explicitly in the default_args even though the same GCP connection is shared throughout the DAG. We propose to: standardize all connection id names, e.g., big_query_conn_id ---> gcp_conn_id google_cloud_conn_id-->gcp_conn_id standardize all default values, e.g., 'bigquery_default' --> 'google_cloud_default' Therefore, if the same GCP connection is used, we only need to specify once in the DAG default_args, e.g., default_args = { ... gcp_conn_id='some_gcp_connection_id' ... } Better still, if a connection with the default name 'google_cloud_default' has already been created and used by all GCP operators, the gcp_conn_id doesn't even need to be specified in DAG default_args. Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)