nathadfield opened a new issue #9897:
URL: https://github.com/apache/airflow/issues/9897
**Apache Airflow version**: 1.10.*
**Environment**:
- **Cloud provider or hardware configuration**: GCP
- **OS** (e.g. from /etc/os-release):
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
**What happened**:
When using the `BigQueryToGCSOperator` in
`providers.google.cloud.transfers.bigquery_to_gcs` with a task defined as
below, despite the task running successfully, no file appears in the GCS bucket
and it doesn’t appear to create a BigQuery jobid; just lots of deprecation
warnings.
> bqtogcs = BigQueryToGCSOperator(
> task_id='bqtest',
> gcp_conn_id=my_gcp_connid,
> source_project_dataset_table='my-project.my_dataset.my_table',
> destination_cloud_storage_uris=['gs://my_bucket/my_file.csv'],
> export_format='CSV',
> print_header=False
> )
>
>
> airflow@540d9822d645:~$ airflow test bqtogcs_test bqtest 2020-07-19
> [2020-07-20 15:20:34,996] {{__init__.py:50}} INFO - Using executor
LocalExecutor
> [2020-07-20 15:20:34,999] {{dagbag.py:396}} INFO - Filling up the DagBag
from /usr/local/airflow/dags
> [2020-07-20 15:20:49,302] {{taskinstance.py:670}} INFO - Dependencies all
met for <TaskInstance: bqtogcs_test.bqtest 2020-07-19T00:00:00+00:00 [None]>
> [2020-07-20 15:20:49,314] {{taskinstance.py:880}} INFO -
> [2020-07-20 15:20:49,314] {{taskinstance.py:881}} INFO - Starting attempt
1 of 4
> [2020-07-20 15:20:49,314] {{taskinstance.py:882}} INFO -
> [2020-07-20 15:20:49,317] {{taskinstance.py:901}} INFO - Executing
<Task(BigQueryToGCSOperator): bqtest> on 2020-07-19T00:00:00+00:00
> [2020-07-20 15:20:49,344] {{bigquery_to_gcs.py:112}} INFO - Executing
extract of my-project.my_dataset.my_table into: ['gs://my_bucket/']
>
/usr/local/lib/python3.7/site-packages/airflow/providers/google/cloud/transfers/bigquery_to_gcs.py:115:
DeprecationWarning: The bigquery_conn_id parameter has been deprecated. You
should pass the gcp_conn_id parameter.
> location=self.location)
>
/usr/local/lib/python3.7/site-packages/airflow/providers/google/cloud/hooks/bigquery.py:103:
DeprecationWarning: This method will be deprecated. Please use
`BigQueryHook.get_client` method
> DeprecationWarning
> /usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py:984:
DeprecationWarning: This method is deprecated. Please use
`airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_extract`
> result = task_copy.execute(context=context)
>
/usr/local/lib/python3.7/site-packages/airflow/providers/google/cloud/hooks/bigquery.py:1890:
DeprecationWarning: This method is deprecated. Please use
`BigQueryHook.insert_job` method.
> DeprecationWarning
> [2020-07-20 15:20:49,739] {{taskinstance.py:1070}} INFO - Marking task as
SUCCESS.dag_id=bqtogcs_test, task_id=bqtest, execution_date=20200719T000000,
start_date=20200720T152049, end_date=20200720T152049
**What you expected to happen**:
I expected to see a CSV file in the desired GCS bucket.
I think its because the job doesn't actually get run. Looking at other
methods, I think that there should also be a `job.result()` here.
https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/bigquery.py#L1924
**How to reproduce it**:
Simply create a task based on this operator as defined above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
