[jira] [Commented] (AIRFLOW-3571) GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS to BiqQuery but a task is failed
[
https://issues.apache.org/jira/browse/AIRFLOW-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730970#comment-16730970
]
jack commented on AIRFLOW-3571:
---
[~yohei] yep. You can create a ticket for all operators (example:
https://issues.apache.org/jira/browse/AIRFLOW-3278 )
> GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS
> to BiqQuery but a task is failed
> -
>
> Key: AIRFLOW-3571
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3571
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.0
>Reporter: Yohei Onishi
>Assignee: Yohei Onishi
>Priority: Major
>
> I am using the following service in asia-northeast1-c zone. * GCS:
> asia-northeast1-c
> * BigQuery dataset and table: asia-northeast1-c
> * Composer: asia-northeast1-c
> My task created by GoogleCloudStorageToBigQueryOperator succeeded to
> uploading CSV file from a GCS bucket to a BigQuery table but the task was
> failed due to the following error.
>
> {code:java}
> [2018-12-26 21:35:47,464] {base_task_runner.py:107} INFO - Job 146: Subtask
> bq_load_data_into_dest_table_from_gcs [2018-12-26 21:35:47,464]
> {discovery.py:871} INFO - URL being requested: GET
> https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> [2018-12-26 21:35:47,931] {models.py:1736} ERROR - ('BigQuery job status
> check failed. Final error was: %s', 404)
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 981, in run_with_configuratio
> jobId=self.running_job_id).execute(
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrappe
> return wrapped(*args, **kwargs
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line
> 851, in execut
> raise HttpError(resp, content, uri=self.uri
> googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> returned "Not found: Job my-project:job_abc123"
> During handling of the above exception, another exception occurred
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/models.py", line 1633, in _run_raw_tas
> result = task_copy.execute(context=context
> File "/usr/local/lib/airflow/airflow/contrib/operators/gcs_to_bq.py", line
> 237, in execut
> time_partitioning=self.time_partitioning
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 951, in run_loa
> return self.run_with_configuration(configuration
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 1003, in run_with_configuratio
> err.resp.status
> Exception: ('BigQuery job status check failed. Final error was: %s', 404
> {code}
> The task failed to find a job {color:#ff}fmy-project:job_abc123{color}
> but the correct job id is{color:#ff}
> my-project:asia-northeast1:job_abc123{color}. (Note: this is just an example,
> not actual id.)
> I suppose the operator does not treat zone properly.
>
> {code:java}
> $ bq show -j my-project:asia-northeast1:job_abc123
> Job my-project:asia-northeast1:job_abc123
> Job Type State Start Time Duration User Email Bytes Processed Bytes Billed
> Billing Tier Labels
> -- - - --
> --
> - -- --
> load SUCCESS 27 Dec 05:35:47 0:00:01 my-service-account-id-email
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3571) GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS to BiqQuery but a task is failed
[
https://issues.apache.org/jira/browse/AIRFLOW-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730967#comment-16730967
]
Yohei Onishi commented on AIRFLOW-3571:
---
[https://github.com/apache/incubator-airflow/pull/4324]
4324 is merged so I can work on my issue.
> GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS
> to BiqQuery but a task is failed
> -
>
> Key: AIRFLOW-3571
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3571
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.0
>Reporter: Yohei Onishi
>Assignee: Yohei Onishi
>Priority: Major
>
> I am using the following service in asia-northeast1-c zone. * GCS:
> asia-northeast1-c
> * BigQuery dataset and table: asia-northeast1-c
> * Composer: asia-northeast1-c
> My task created by GoogleCloudStorageToBigQueryOperator succeeded to
> uploading CSV file from a GCS bucket to a BigQuery table but the task was
> failed due to the following error.
>
> {code:java}
> [2018-12-26 21:35:47,464] {base_task_runner.py:107} INFO - Job 146: Subtask
> bq_load_data_into_dest_table_from_gcs [2018-12-26 21:35:47,464]
> {discovery.py:871} INFO - URL being requested: GET
> https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> [2018-12-26 21:35:47,931] {models.py:1736} ERROR - ('BigQuery job status
> check failed. Final error was: %s', 404)
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 981, in run_with_configuratio
> jobId=self.running_job_id).execute(
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrappe
> return wrapped(*args, **kwargs
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line
> 851, in execut
> raise HttpError(resp, content, uri=self.uri
> googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> returned "Not found: Job my-project:job_abc123"
> During handling of the above exception, another exception occurred
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/models.py", line 1633, in _run_raw_tas
> result = task_copy.execute(context=context
> File "/usr/local/lib/airflow/airflow/contrib/operators/gcs_to_bq.py", line
> 237, in execut
> time_partitioning=self.time_partitioning
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 951, in run_loa
> return self.run_with_configuration(configuration
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 1003, in run_with_configuratio
> err.resp.status
> Exception: ('BigQuery job status check failed. Final error was: %s', 404
> {code}
> The task failed to find a job {color:#ff}fmy-project:job_abc123{color}
> but the correct job id is{color:#ff}
> my-project:asia-northeast1:job_abc123{color}. (Note: this is just an example,
> not actual id.)
> I suppose the operator does not treat zone properly.
>
> {code:java}
> $ bq show -j my-project:asia-northeast1:job_abc123
> Job my-project:asia-northeast1:job_abc123
> Job Type State Start Time Duration User Email Bytes Processed Bytes Billed
> Billing Tier Labels
> -- - - --
> --
> - -- --
> load SUCCESS 27 Dec 05:35:47 0:00:01 my-service-account-id-email
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3571) GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS to BiqQuery but a task is failed
[
https://issues.apache.org/jira/browse/AIRFLOW-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730906#comment-16730906
]
Yohei Onishi commented on AIRFLOW-3571:
---
OK will do
> GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS
> to BiqQuery but a task is failed
> -
>
> Key: AIRFLOW-3571
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3571
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.0
>Reporter: Yohei Onishi
>Assignee: Yohei Onishi
>Priority: Major
>
> I am using the following service in asia-northeast1-c zone. * GCS:
> asia-northeast1-c
> * BigQuery dataset and table: asia-northeast1-c
> * Composer: asia-northeast1-c
> My task created by GoogleCloudStorageToBigQueryOperator succeeded to
> uploading CSV file from a GCS bucket to a BigQuery table but the task was
> failed due to the following error.
>
> {code:java}
> [2018-12-26 21:35:47,464] {base_task_runner.py:107} INFO - Job 146: Subtask
> bq_load_data_into_dest_table_from_gcs [2018-12-26 21:35:47,464]
> {discovery.py:871} INFO - URL being requested: GET
> https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> [2018-12-26 21:35:47,931] {models.py:1736} ERROR - ('BigQuery job status
> check failed. Final error was: %s', 404)
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 981, in run_with_configuratio
> jobId=self.running_job_id).execute(
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrappe
> return wrapped(*args, **kwargs
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line
> 851, in execut
> raise HttpError(resp, content, uri=self.uri
> googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> returned "Not found: Job my-project:job_abc123"
> During handling of the above exception, another exception occurred
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/models.py", line 1633, in _run_raw_tas
> result = task_copy.execute(context=context
> File "/usr/local/lib/airflow/airflow/contrib/operators/gcs_to_bq.py", line
> 237, in execut
> time_partitioning=self.time_partitioning
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 951, in run_loa
> return self.run_with_configuration(configuration
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 1003, in run_with_configuratio
> err.resp.status
> Exception: ('BigQuery job status check failed. Final error was: %s', 404
> {code}
> The task failed to find a job {color:#ff}fmy-project:job_abc123{color}
> but the correct job id is{color:#ff}
> my-project:asia-northeast1:job_abc123{color}. (Note: this is just an example,
> not actual id.)
> I suppose the operator does not treat zone properly.
>
> {code:java}
> $ bq show -j my-project:asia-northeast1:job_abc123
> Job my-project:asia-northeast1:job_abc123
> Job Type State Start Time Duration User Email Bytes Processed Bytes Billed
> Billing Tier Labels
> -- - - --
> --
> - -- --
> load SUCCESS 27 Dec 05:35:47 0:00:01 my-service-account-id-email
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3571) GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS to BiqQuery but a task is failed
[
https://issues.apache.org/jira/browse/AIRFLOW-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730064#comment-16730064
]
Yohei Onishi commented on AIRFLOW-3571:
---
Sorry I misunderstood the PR. This just fixes
[bigquery_hook.py|https://github.com/apache/incubator-airflow/pull/4324/files#diff-ee06f8fcbc476ea65446a30160c2a2b2]
and [http://bigquery_operator.py|http://bigquery_operator.py./] only. Not
GoogleCloudStorageToBigQueryOperator
[https://github.com/apache/incubator-airflow/pull/4324]
> GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS
> to BiqQuery but a task is failed
> -
>
> Key: AIRFLOW-3571
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3571
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.0
>Reporter: Yohei Onishi
>Assignee: Yohei Onishi
>Priority: Major
>
> I am using the following service in asia-northeast1-c zone. * GCS:
> asia-northeast1-c
> * BigQuery dataset and table: asia-northeast1-c
> * Composer: asia-northeast1-c
> My task created by GoogleCloudStorageToBigQueryOperator succeeded to
> uploading CSV file from a GCS bucket to a BigQuery table but the task was
> failed due to the following error.
>
> {code:java}
> [2018-12-26 21:35:47,464] {base_task_runner.py:107} INFO - Job 146: Subtask
> bq_load_data_into_dest_table_from_gcs [2018-12-26 21:35:47,464]
> {discovery.py:871} INFO - URL being requested: GET
> https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> [2018-12-26 21:35:47,931] {models.py:1736} ERROR - ('BigQuery job status
> check failed. Final error was: %s', 404)
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 981, in run_with_configuratio
> jobId=self.running_job_id).execute(
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrappe
> return wrapped(*args, **kwargs
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line
> 851, in execut
> raise HttpError(resp, content, uri=self.uri
> googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> returned "Not found: Job my-project:job_abc123"
> During handling of the above exception, another exception occurred
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/models.py", line 1633, in _run_raw_tas
> result = task_copy.execute(context=context
> File "/usr/local/lib/airflow/airflow/contrib/operators/gcs_to_bq.py", line
> 237, in execut
> time_partitioning=self.time_partitioning
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 951, in run_loa
> return self.run_with_configuration(configuration
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 1003, in run_with_configuratio
> err.resp.status
> Exception: ('BigQuery job status check failed. Final error was: %s', 404
> {code}
> The task failed to find a job {color:#ff}fmy-project:job_abc123{color}
> but the correct job id is{color:#ff}
> my-project:asia-northeast1:job_abc123{color}. (Note: this is just an example,
> not actual id.)
> I suppose the operator does not treat zone properly.
>
> {code:java}
> $ bq show -j my-project:asia-northeast1:job_abc123
> Job my-project:asia-northeast1:job_abc123
> Job Type State Start Time Duration User Email Bytes Processed Bytes Billed
> Billing Tier Labels
> -- - - --
> --
> - -- --
> load SUCCESS 27 Dec 05:35:47 0:00:01 my-service-account-id-email
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3571) GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS to BiqQuery but a task is failed
[
https://issues.apache.org/jira/browse/AIRFLOW-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729468#comment-16729468
]
Yohei Onishi commented on AIRFLOW-3571:
---
This PR is also fixing the operator as well so I will close once the PR is
merged.
> GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS
> to BiqQuery but a task is failed
> -
>
> Key: AIRFLOW-3571
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3571
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.0
>Reporter: Yohei Onishi
>Assignee: Yohei Onishi
>Priority: Major
>
> I am using the following service in asia-northeast1-c zone. * GCS:
> asia-northeast1-c
> * BigQuery dataset and table: asia-northeast1-c
> * Composer: asia-northeast1-c
> My task created by GoogleCloudStorageToBigQueryOperator succeeded to
> uploading CSV file from a GCS bucket to a BigQuery table but the task was
> failed due to the following error.
>
> {code:java}
> [2018-12-26 21:35:47,464] {base_task_runner.py:107} INFO - Job 146: Subtask
> bq_load_data_into_dest_table_from_gcs [2018-12-26 21:35:47,464]
> {discovery.py:871} INFO - URL being requested: GET
> https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> [2018-12-26 21:35:47,931] {models.py:1736} ERROR - ('BigQuery job status
> check failed. Final error was: %s', 404)
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 981, in run_with_configuratio
> jobId=self.running_job_id).execute(
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrappe
> return wrapped(*args, **kwargs
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line
> 851, in execut
> raise HttpError(resp, content, uri=self.uri
> googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> returned "Not found: Job my-project:job_abc123"
> During handling of the above exception, another exception occurred
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/models.py", line 1633, in _run_raw_tas
> result = task_copy.execute(context=context
> File "/usr/local/lib/airflow/airflow/contrib/operators/gcs_to_bq.py", line
> 237, in execut
> time_partitioning=self.time_partitioning
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 951, in run_loa
> return self.run_with_configuration(configuration
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 1003, in run_with_configuratio
> err.resp.status
> Exception: ('BigQuery job status check failed. Final error was: %s', 404
> {code}
> The task failed to find a job {color:#ff}fmy-project:job_abc123{color}
> but the correct job id is{color:#ff}
> my-project:asia-northeast1:job_abc123{color}. (Note: this is just an example,
> not actual id.)
> I suppose the operator does not treat zone properly.
>
> {code:java}
> $ bq show -j my-project:asia-northeast1:job_abc123
> Job my-project:asia-northeast1:job_abc123
> Job Type State Start Time Duration User Email Bytes Processed Bytes Billed
> Billing Tier Labels
> -- - - --
> --
> - -- --
> load SUCCESS 27 Dec 05:35:47 0:00:01 my-service-account-id-email
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3571) GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS to BiqQuery but a task is failed
[
https://issues.apache.org/jira/browse/AIRFLOW-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729399#comment-16729399
]
Yohei Onishi commented on AIRFLOW-3571:
---
Thanks, will do.
> GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS
> to BiqQuery but a task is failed
> -
>
> Key: AIRFLOW-3571
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3571
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.0
>Reporter: Yohei Onishi
>Assignee: Yohei Onishi
>Priority: Major
>
> I am using the following service in asia-northeast1-c zone. * GCS:
> asia-northeast1-c
> * BigQuery dataset and table: asia-northeast1-c
> * Composer: asia-northeast1-c
> My task created by GoogleCloudStorageToBigQueryOperator succeeded to
> uploading CSV file from a GCS bucket to a BigQuery table but the task was
> failed due to the following error.
>
> {code:java}
> [2018-12-26 21:35:47,464] {base_task_runner.py:107} INFO - Job 146: Subtask
> bq_load_data_into_dest_table_from_gcs [2018-12-26 21:35:47,464]
> {discovery.py:871} INFO - URL being requested: GET
> https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> [2018-12-26 21:35:47,931] {models.py:1736} ERROR - ('BigQuery job status
> check failed. Final error was: %s', 404)
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 981, in run_with_configuratio
> jobId=self.running_job_id).execute(
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrappe
> return wrapped(*args, **kwargs
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line
> 851, in execut
> raise HttpError(resp, content, uri=self.uri
> googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> returned "Not found: Job my-project:job_abc123"
> During handling of the above exception, another exception occurred
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/models.py", line 1633, in _run_raw_tas
> result = task_copy.execute(context=context
> File "/usr/local/lib/airflow/airflow/contrib/operators/gcs_to_bq.py", line
> 237, in execut
> time_partitioning=self.time_partitioning
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 951, in run_loa
> return self.run_with_configuration(configuration
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 1003, in run_with_configuratio
> err.resp.status
> Exception: ('BigQuery job status check failed. Final error was: %s', 404
> {code}
> The task failed to find a job {color:#ff}fmy-project:job_abc123{color}
> but the correct job id is{color:#ff}
> my-project:asia-northeast1:job_abc123{color}. (Note: this is just an example,
> not actual id.)
> I suppose the operator does not treat zone properly.
>
> {code:java}
> $ bq show -j my-project:asia-northeast1:job_abc123
> Job my-project:asia-northeast1:job_abc123
> Job Type State Start Time Duration User Email Bytes Processed Bytes Billed
> Billing Tier Labels
> -- - - --
> --
> - -- --
> load SUCCESS 27 Dec 05:35:47 0:00:01 my-service-account-id-email
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3571) GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS to BiqQuery but a task is failed
[
https://issues.apache.org/jira/browse/AIRFLOW-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729398#comment-16729398
]
jack commented on AIRFLOW-3571:
---
[~yohei] There is an open PR for adding support for locations to the
BigQueryHook https://github.com/apache/incubator-airflow/pull/4324
Once it's merged (I assume it would be soon) you are welcome to open PRs for
extending the Operators to support it.
> GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS
> to BiqQuery but a task is failed
> -
>
> Key: AIRFLOW-3571
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3571
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.0
>Reporter: Yohei Onishi
>Priority: Major
>
> I am using the following service in asia-northeast1-c zone. * GCS:
> asia-northeast1-c
> * BigQuery dataset and table: asia-northeast1-c
> * Composer: asia-northeast1-c
> My task created by GoogleCloudStorageToBigQueryOperator succeeded to
> uploading CSV file from a GCS bucket to a BigQuery table but the task was
> failed due to the following error.
>
> {code:java}
> [2018-12-26 21:35:47,464] {base_task_runner.py:107} INFO - Job 146: Subtask
> bq_load_data_into_dest_table_from_gcs [2018-12-26 21:35:47,464]
> {discovery.py:871} INFO - URL being requested: GET
> https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> [2018-12-26 21:35:47,931] {models.py:1736} ERROR - ('BigQuery job status
> check failed. Final error was: %s', 404)
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 981, in run_with_configuratio
> jobId=self.running_job_id).execute(
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrappe
> return wrapped(*args, **kwargs
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line
> 851, in execut
> raise HttpError(resp, content, uri=self.uri
> googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> returned "Not found: Job my-project:job_abc123"
> During handling of the above exception, another exception occurred
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/models.py", line 1633, in _run_raw_tas
> result = task_copy.execute(context=context
> File "/usr/local/lib/airflow/airflow/contrib/operators/gcs_to_bq.py", line
> 237, in execut
> time_partitioning=self.time_partitioning
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 951, in run_loa
> return self.run_with_configuration(configuration
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 1003, in run_with_configuratio
> err.resp.status
> Exception: ('BigQuery job status check failed. Final error was: %s', 404
> {code}
> The task failed to find a job {color:#ff}fmy-project:job_abc123{color}
> but the correct job id is{color:#ff}
> my-project:asia-northeast1:job_abc123{color}. (Note: this is just an example,
> not actual id.)
> I suppose the operator does not treat zone properly.
>
> {code:java}
> $ bq show -j my-project:asia-northeast1:job_abc123
> Job my-project:asia-northeast1:job_abc123
> Job Type State Start Time Duration User Email Bytes Processed Bytes Billed
> Billing Tier Labels
> -- - - --
> --
> - -- --
> load SUCCESS 27 Dec 05:35:47 0:00:01 my-service-account-id-email
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3571) GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS to BiqQuery but a task is failed
[
https://issues.apache.org/jira/browse/AIRFLOW-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729263#comment-16729263
]
Yohei Onishi commented on AIRFLOW-3571:
---
It seems {color:#d04437}GoogleCloudStorageToBigQueryOperator{color} does not
support regions other than US / EU.
> GoogleCloudStorageToBigQueryOperator succeeds to uploading CSV file from GCS
> to BiqQuery but a task is failed
> -
>
> Key: AIRFLOW-3571
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3571
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.0
>Reporter: Yohei Onishi
>Priority: Major
>
> I am using the following service in asia-northeast1-c zone. * GCS:
> asia-northeast1-c
> * BigQuery dataset and table: asia-northeast1-c
> * Composer: asia-northeast1-c
> My task created by GoogleCloudStorageToBigQueryOperator succeeded to
> uploading CSV file from a GCS bucket to a BigQuery table but the task was
> failed due to the following error.
>
> {code:java}
> [2018-12-26 21:35:47,464] {base_task_runner.py:107} INFO - Job 146: Subtask
> bq_load_data_into_dest_table_from_gcs [2018-12-26 21:35:47,464]
> {discovery.py:871} INFO - URL being requested: GET
> https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> [2018-12-26 21:35:47,931] {models.py:1736} ERROR - ('BigQuery job status
> check failed. Final error was: %s', 404)
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 981, in run_with_configuratio
> jobId=self.running_job_id).execute(
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrappe
> return wrapped(*args, **kwargs
> File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line
> 851, in execut
> raise HttpError(resp, content, uri=self.uri
> googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_abc123?alt=json
> returned "Not found: Job my-project:job_abc123"
> During handling of the above exception, another exception occurred
> Traceback (most recent call last)
> File "/usr/local/lib/airflow/airflow/models.py", line 1633, in _run_raw_tas
> result = task_copy.execute(context=context
> File "/usr/local/lib/airflow/airflow/contrib/operators/gcs_to_bq.py", line
> 237, in execut
> time_partitioning=self.time_partitioning
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 951, in run_loa
> return self.run_with_configuration(configuration
> File "/usr/local/lib/airflow/airflow/contrib/hooks/bigquery_hook.py", line
> 1003, in run_with_configuratio
> err.resp.status
> Exception: ('BigQuery job status check failed. Final error was: %s', 404
> {code}
> The task failed to find a job {color:#ff}fmy-project:job_abc123{color}
> but the correct job id is{color:#ff}
> my-project:asia-northeast1:job_abc123{color}. (Note: this is just an example,
> not actual id.)
> I suppose the operator does not treat zone properly.
>
> {code:java}
> $ bq show -j my-project:asia-northeast1:job_abc123
> Job my-project:asia-northeast1:job_abc123
> Job Type State Start Time Duration User Email Bytes Processed Bytes Billed
> Billing Tier Labels
> -- - - --
> --
> - -- --
> load SUCCESS 27 Dec 05:35:47 0:00:01 my-service-account-id-email
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
