bhirsz commented on a change in pull request #22311: URL: https://github.com/apache/airflow/pull/22311#discussion_r829326254
########## File path: tests/system/providers/google/bigquery/example_bigquery_dataset.py ########## @@ -0,0 +1,94 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Example Airflow DAG for Google BigQuery service testing dataset operations. +""" +import os +from datetime import datetime + +from airflow import models +from airflow.operators.bash import BashOperator +from airflow.providers.google.cloud.operators.bigquery import ( + BigQueryCreateEmptyDatasetOperator, + BigQueryDeleteDatasetOperator, + BigQueryGetDatasetOperator, + BigQueryUpdateDatasetOperator, +) +from airflow.utils.trigger_rule import TriggerRule Review comment: It's not - we're using trigger rules only if there is teardown-like task. For example if you're creating resource using one task you want to make sure to run task that removes that resource even if some tasks inbetween fails. There may be system tests that don't need it at all. ########## File path: scripts/ci/pre_commit/pre_commit_check_watcher_in_examples.py ########## @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import sys +from pathlib import Path +from typing import List + +from rich.console import Console + +if __name__ not in ("__main__", "__mp_main__"): + raise SystemExit( + "This file is intended to be executed as an executable program. You cannot use it as a module." + f"To run this script, run the ./{__file__} command [FILE] ..." + ) + + +console = Console(color_system="standard", width=200) + +errors: List[str] = [] + +WATCHER_APPEND_INSTRUCTION = "list(dag.tasks) >> watcher()" + +PYTEST_FUNCTION = """ +def test_run(): Review comment: Unfortunately yes. Although in this case it isn't complicated to replace few lines - one file or 100 it doesn't matter for your awk/sed script ;). It's bit of the trade off - we could also leave example dags without any pytest method and generate tests dynamically but by adding this method it's easier to run just one system tests. ########## File path: tests/system/providers/google/bigquery/example_bigquery_operations.py ########## @@ -0,0 +1,106 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Example Airflow DAG for Google BigQuery service local file upload and external table creation. +""" +import os +from datetime import datetime +from pathlib import Path + +from airflow import models +from airflow.providers.google.cloud.operators.bigquery import ( + BigQueryCreateEmptyDatasetOperator, + BigQueryCreateExternalTableOperator, + BigQueryDeleteDatasetOperator, +) +from airflow.providers.google.cloud.operators.gcs import GCSCreateBucketOperator, GCSDeleteBucketOperator +from airflow.providers.google.cloud.transfers.local_to_gcs import LocalFilesystemToGCSOperator +from airflow.utils.trigger_rule import TriggerRule +from tests.system.utils.watcher import watcher + +ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID") +DAG_ID = "bigquery_operations" + +DATASET_NAME = f"dataset_{DAG_ID}_{ENV_ID}" +DATA_SAMPLE_GCS_BUCKET_NAME = f"bucket_{DAG_ID}_{ENV_ID}" +DATA_SAMPLE_GCS_OBJECT_NAME = "bigquery/us-states/us-states.csv" +CSV_FILE_LOCAL_PATH = str(Path(__file__).parent / "resources" / "us-states.csv") + + +with models.DAG( + DAG_ID, + schedule_interval="@once", + start_date=datetime(2021, 1, 1), + catchup=False, + tags=["example", "bigquery"], +) as dag: + create_bucket = GCSCreateBucketOperator(task_id="create_bucket", bucket_name=DATA_SAMPLE_GCS_BUCKET_NAME) + + create_dataset = BigQueryCreateEmptyDatasetOperator(task_id="create_dataset", dataset_id=DATASET_NAME) + + upload_file = LocalFilesystemToGCSOperator( + task_id="upload_file_to_bucket", + src=CSV_FILE_LOCAL_PATH, + dst=DATA_SAMPLE_GCS_OBJECT_NAME, + bucket=DATA_SAMPLE_GCS_BUCKET_NAME, + ) + + # [START howto_operator_bigquery_create_external_table] + create_external_table = BigQueryCreateExternalTableOperator( + task_id="create_external_table", + destination_project_dataset_table=f"{DATASET_NAME}.external_table", + bucket=DATA_SAMPLE_GCS_BUCKET_NAME, + source_objects=[DATA_SAMPLE_GCS_OBJECT_NAME], + schema_fields=[ + {"name": "emp_name", "type": "STRING", "mode": "REQUIRED"}, + {"name": "salary", "type": "INTEGER", "mode": "NULLABLE"}, + ], + ) + # [END howto_operator_bigquery_create_external_table] + + delete_dataset = BigQueryDeleteDatasetOperator( + task_id="delete_dataset", + dataset_id=DATASET_NAME, + delete_contents=True, + trigger_rule=TriggerRule.ALL_DONE, + ) + + delete_bucket = GCSDeleteBucketOperator( + task_id="delete_bucket", bucket_name=DATA_SAMPLE_GCS_BUCKET_NAME, trigger_rule=TriggerRule.ALL_DONE + ) + + ( + # TEST SETUP + [create_bucket, create_dataset] + # TEST BODY + >> upload_file + >> create_external_table + # TEST TEARDOWN + >> delete_dataset + >> delete_bucket + ) + + list(dag.tasks) >> watcher() Review comment: > Writing the same line in hundreds of files just feels wrong I fully agree with you and it was actually the reason for the redesign - where we had dozens of lines repeated for every test only to trigger other file with the DAG. And we always welcome any suggestions and discussion. We changed the design several times over the time thanks for the input from the others. I agree with Jarek here - the goal is to have an example dag that can be used by other Airflow users as reference and it would be best to avoid any code specific only to system test. Proposed local import of the watcher looks like a good idea for me for this reason alone ########## File path: tests/system/providers/google/bigquery/example_bigquery_operations.py ########## @@ -0,0 +1,106 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Example Airflow DAG for Google BigQuery service local file upload and external table creation. +""" +import os +from datetime import datetime +from pathlib import Path + +from airflow import models +from airflow.providers.google.cloud.operators.bigquery import ( + BigQueryCreateEmptyDatasetOperator, + BigQueryCreateExternalTableOperator, + BigQueryDeleteDatasetOperator, +) +from airflow.providers.google.cloud.operators.gcs import GCSCreateBucketOperator, GCSDeleteBucketOperator +from airflow.providers.google.cloud.transfers.local_to_gcs import LocalFilesystemToGCSOperator +from airflow.utils.trigger_rule import TriggerRule +from tests.system.utils.watcher import watcher + +ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID") +DAG_ID = "bigquery_operations" + +DATASET_NAME = f"dataset_{DAG_ID}_{ENV_ID}" +DATA_SAMPLE_GCS_BUCKET_NAME = f"bucket_{DAG_ID}_{ENV_ID}" +DATA_SAMPLE_GCS_OBJECT_NAME = "bigquery/us-states/us-states.csv" +CSV_FILE_LOCAL_PATH = str(Path(__file__).parent / "resources" / "us-states.csv") + + +with models.DAG( + DAG_ID, + schedule_interval="@once", + start_date=datetime(2021, 1, 1), + catchup=False, + tags=["example", "bigquery"], +) as dag: + create_bucket = GCSCreateBucketOperator(task_id="create_bucket", bucket_name=DATA_SAMPLE_GCS_BUCKET_NAME) + + create_dataset = BigQueryCreateEmptyDatasetOperator(task_id="create_dataset", dataset_id=DATASET_NAME) + + upload_file = LocalFilesystemToGCSOperator( + task_id="upload_file_to_bucket", + src=CSV_FILE_LOCAL_PATH, + dst=DATA_SAMPLE_GCS_OBJECT_NAME, + bucket=DATA_SAMPLE_GCS_BUCKET_NAME, + ) + + # [START howto_operator_bigquery_create_external_table] + create_external_table = BigQueryCreateExternalTableOperator( + task_id="create_external_table", + destination_project_dataset_table=f"{DATASET_NAME}.external_table", + bucket=DATA_SAMPLE_GCS_BUCKET_NAME, + source_objects=[DATA_SAMPLE_GCS_OBJECT_NAME], + schema_fields=[ + {"name": "emp_name", "type": "STRING", "mode": "REQUIRED"}, + {"name": "salary", "type": "INTEGER", "mode": "NULLABLE"}, + ], + ) + # [END howto_operator_bigquery_create_external_table] + + delete_dataset = BigQueryDeleteDatasetOperator( + task_id="delete_dataset", + dataset_id=DATASET_NAME, + delete_contents=True, + trigger_rule=TriggerRule.ALL_DONE, + ) + + delete_bucket = GCSDeleteBucketOperator( + task_id="delete_bucket", bucket_name=DATA_SAMPLE_GCS_BUCKET_NAME, trigger_rule=TriggerRule.ALL_DONE + ) + + ( + # TEST SETUP + [create_bucket, create_dataset] + # TEST BODY + >> upload_file + >> create_external_table + # TEST TEARDOWN + >> delete_dataset + >> delete_bucket + ) + + list(dag.tasks) >> watcher() Review comment: > Writing the same line in hundreds of files just feels wrong I fully agree with you and it was actually the reason for the redesign - where we had dozens of lines repeated for every test only to trigger other file with the DAG. And we always welcome any suggestions and discussion. We changed the design several times over the time thanks for the input from the others. I agree with Jarek here - the goal is to have an example dag that can be used by other Airflow users as reference and it would be best to avoid any code specific only to system test. Proposed local import of the watcher looks like a good idea for me for this reason alone ########## File path: scripts/ci/pre_commit/pre_commit_check_watcher_in_examples.py ########## @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import sys +from pathlib import Path +from typing import List + +from rich.console import Console + +if __name__ not in ("__main__", "__mp_main__"): + raise SystemExit( + "This file is intended to be executed as an executable program. You cannot use it as a module." + f"To run this script, run the ./{__file__} command [FILE] ..." + ) + + +console = Console(color_system="standard", width=200) + +errors: List[str] = [] + +WATCHER_APPEND_INSTRUCTION = "list(dag.tasks) >> watcher()" + +PYTEST_FUNCTION = """ +def test_run(): Review comment: I like the idea - it's also making it easier for temporary updates of the test method (ie adding verbose mode, overwriting executor etc). I've tested it and it appears to be working. Pytest recognizes it if I run it using pytest command however it's not detected by IDE as test method (but maybe I can workaround it somehow with pytest ini). I will discuss it with @mnojek and if we both agree we can update it ########## File path: tests/system/conftest.py ########## @@ -0,0 +1,61 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import os +import re +from itertools import chain +from pathlib import Path + +import pytest + +OLD_EXECUTOR = os.environ.get("AIRFLOW__CORE__EXECUTOR", default=None) +REQUIRED_ENV_VARS = ("SYSTEM_TESTS_ENV_ID",) + + +def pytest_configure(config): + os.environ["AIRFLOW__CORE__EXECUTOR"] = "DebugExecutor" + + +def pytest_unconfigure(config): + if OLD_EXECUTOR is not None: + os.environ["AIRFLOW__CORE__EXECUTOR"] = OLD_EXECUTOR Review comment: I think that with the AIRFLOW__CORE__EXECUTOR set to None the execution will fail anyway. But I could introduce extra flag ie: ``` IS_EXECUTOR_SET = "AIRFLOW__CORE__EXECUTOR" in os.environ OLD_EXECUTOR = os.environ.get("AIRFLOW__CORE__EXECUTOR", default=None) (...) def pytest_unconfigure(config): if IS_EXECUTOR_SET: os.environ["AIRFLOW__CORE__EXECUTOR"] = OLD_EXECUTOR ``` ########## File path: docs/apache-airflow-providers-google/operators/cloud/bigquery.rst ########## @@ -29,7 +29,7 @@ data. Prerequisite Tasks ^^^^^^^^^^^^^^^^^^ -.. include::/operators/_partials/prerequisite_tasks.rst +.. include:: ../_partials/prerequisite_tasks.rst Review comment: It's not although it's common practice, see sphinx docs: https://docutils.sourceforge.io/docs/ref/rst/directives.html#include . This line is actually not needed for system tests - it's fix because I've noticed that Google provider docs doesn't contain prerequisite_tasks section content (due to invalid include - for at least few versions). It's very minor so I think it's fine to also include this change in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
