potiuk commented on a change in pull request #22311: URL: https://github.com/apache/airflow/pull/22311#discussion_r829580834
########## File path: tests/system/providers/google/bigquery/example_bigquery_operations.py ########## @@ -0,0 +1,106 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Example Airflow DAG for Google BigQuery service local file upload and external table creation. +""" +import os +from datetime import datetime +from pathlib import Path + +from airflow import models +from airflow.providers.google.cloud.operators.bigquery import ( + BigQueryCreateEmptyDatasetOperator, + BigQueryCreateExternalTableOperator, + BigQueryDeleteDatasetOperator, +) +from airflow.providers.google.cloud.operators.gcs import GCSCreateBucketOperator, GCSDeleteBucketOperator +from airflow.providers.google.cloud.transfers.local_to_gcs import LocalFilesystemToGCSOperator +from airflow.utils.trigger_rule import TriggerRule +from tests.system.utils.watcher import watcher + +ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID") +DAG_ID = "bigquery_operations" + +DATASET_NAME = f"dataset_{DAG_ID}_{ENV_ID}" +DATA_SAMPLE_GCS_BUCKET_NAME = f"bucket_{DAG_ID}_{ENV_ID}" +DATA_SAMPLE_GCS_OBJECT_NAME = "bigquery/us-states/us-states.csv" +CSV_FILE_LOCAL_PATH = str(Path(__file__).parent / "resources" / "us-states.csv") + + +with models.DAG( + DAG_ID, + schedule_interval="@once", + start_date=datetime(2021, 1, 1), + catchup=False, + tags=["example", "bigquery"], +) as dag: + create_bucket = GCSCreateBucketOperator(task_id="create_bucket", bucket_name=DATA_SAMPLE_GCS_BUCKET_NAME) + + create_dataset = BigQueryCreateEmptyDatasetOperator(task_id="create_dataset", dataset_id=DATASET_NAME) + + upload_file = LocalFilesystemToGCSOperator( + task_id="upload_file_to_bucket", + src=CSV_FILE_LOCAL_PATH, + dst=DATA_SAMPLE_GCS_OBJECT_NAME, + bucket=DATA_SAMPLE_GCS_BUCKET_NAME, + ) + + # [START howto_operator_bigquery_create_external_table] + create_external_table = BigQueryCreateExternalTableOperator( + task_id="create_external_table", + destination_project_dataset_table=f"{DATASET_NAME}.external_table", + bucket=DATA_SAMPLE_GCS_BUCKET_NAME, + source_objects=[DATA_SAMPLE_GCS_OBJECT_NAME], + schema_fields=[ + {"name": "emp_name", "type": "STRING", "mode": "REQUIRED"}, + {"name": "salary", "type": "INTEGER", "mode": "NULLABLE"}, + ], + ) + # [END howto_operator_bigquery_create_external_table] + + delete_dataset = BigQueryDeleteDatasetOperator( + task_id="delete_dataset", + dataset_id=DATASET_NAME, + delete_contents=True, + trigger_rule=TriggerRule.ALL_DONE, + ) + + delete_bucket = GCSDeleteBucketOperator( + task_id="delete_bucket", bucket_name=DATA_SAMPLE_GCS_BUCKET_NAME, trigger_rule=TriggerRule.ALL_DONE + ) + + ( + # TEST SETUP + [create_bucket, create_dataset] + # TEST BODY + >> upload_file + >> create_external_table + # TEST TEARDOWN + >> delete_dataset + >> delete_bucket + ) + + list(dag.tasks) >> watcher() Review comment: We thought about a special "util" function for that. I do not think we discussed about "magical" adding it. this is a bit of a hassle, I agreee but adding it would likely require something on the DAG level - for example a special parameter or decorator. And the problem with that is that it would have to be added somewhere at the beginning of the DAG - where you define the DAG most likely. And the problem with that is that you also use the DAG as examples in our documentation. And we extracts parts of the examples into the documentaiton and we should not pollute those examples with things that are not really good "examples" on how you should add your DAGs. - and those example usually show the DAG definition/default_args as part of the example - by having the special decorator, or parameter on the DAG to indicate that DAG shoudl have "watcher" added migth be too easily copied from those extracted examples. Having explicit watcher makes it so much easier - it is at the and and it is explicit (which means there is no magic) But actually what made me think now - we should actually make it even more separated an explicit. @mnojek why don't we change the the watcher's local import and some comment there to make it even more separated and "explicit" for example: ``` >> delete_bucket ) from tests.system.utils.watcher import watcher # This test run as a system test needs watcher in order to mark success/failure # where "tearDown" task is part of the DAG list(dag.tasks) >> watcher() ``` ########## File path: tests/system/providers/google/bigquery/example_bigquery_operations.py ########## @@ -0,0 +1,106 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Example Airflow DAG for Google BigQuery service local file upload and external table creation. +""" +import os +from datetime import datetime +from pathlib import Path + +from airflow import models +from airflow.providers.google.cloud.operators.bigquery import ( + BigQueryCreateEmptyDatasetOperator, + BigQueryCreateExternalTableOperator, + BigQueryDeleteDatasetOperator, +) +from airflow.providers.google.cloud.operators.gcs import GCSCreateBucketOperator, GCSDeleteBucketOperator +from airflow.providers.google.cloud.transfers.local_to_gcs import LocalFilesystemToGCSOperator +from airflow.utils.trigger_rule import TriggerRule +from tests.system.utils.watcher import watcher + +ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID") +DAG_ID = "bigquery_operations" + +DATASET_NAME = f"dataset_{DAG_ID}_{ENV_ID}" +DATA_SAMPLE_GCS_BUCKET_NAME = f"bucket_{DAG_ID}_{ENV_ID}" +DATA_SAMPLE_GCS_OBJECT_NAME = "bigquery/us-states/us-states.csv" +CSV_FILE_LOCAL_PATH = str(Path(__file__).parent / "resources" / "us-states.csv") + + +with models.DAG( + DAG_ID, + schedule_interval="@once", + start_date=datetime(2021, 1, 1), + catchup=False, + tags=["example", "bigquery"], +) as dag: + create_bucket = GCSCreateBucketOperator(task_id="create_bucket", bucket_name=DATA_SAMPLE_GCS_BUCKET_NAME) + + create_dataset = BigQueryCreateEmptyDatasetOperator(task_id="create_dataset", dataset_id=DATASET_NAME) + + upload_file = LocalFilesystemToGCSOperator( + task_id="upload_file_to_bucket", + src=CSV_FILE_LOCAL_PATH, + dst=DATA_SAMPLE_GCS_OBJECT_NAME, + bucket=DATA_SAMPLE_GCS_BUCKET_NAME, + ) + + # [START howto_operator_bigquery_create_external_table] + create_external_table = BigQueryCreateExternalTableOperator( + task_id="create_external_table", + destination_project_dataset_table=f"{DATASET_NAME}.external_table", + bucket=DATA_SAMPLE_GCS_BUCKET_NAME, + source_objects=[DATA_SAMPLE_GCS_OBJECT_NAME], + schema_fields=[ + {"name": "emp_name", "type": "STRING", "mode": "REQUIRED"}, + {"name": "salary", "type": "INTEGER", "mode": "NULLABLE"}, + ], + ) + # [END howto_operator_bigquery_create_external_table] + + delete_dataset = BigQueryDeleteDatasetOperator( + task_id="delete_dataset", + dataset_id=DATASET_NAME, + delete_contents=True, + trigger_rule=TriggerRule.ALL_DONE, + ) + + delete_bucket = GCSDeleteBucketOperator( + task_id="delete_bucket", bucket_name=DATA_SAMPLE_GCS_BUCKET_NAME, trigger_rule=TriggerRule.ALL_DONE + ) + + ( + # TEST SETUP + [create_bucket, create_dataset] + # TEST BODY + >> upload_file + >> create_external_table + # TEST TEARDOWN + >> delete_dataset + >> delete_bucket + ) + + list(dag.tasks) >> watcher() Review comment: We thought about a special "util" function for that. I do not think we discussed about "magical" adding it. this is a bit of a hassle, I agreee but adding it would likely require something on the DAG level - for example a special parameter or decorator. And the problem with that is that it would have to be added somewhere at the beginning of the DAG - where you define the DAG most likely. And the problem with that is that you also use the DAG as examples in our documentation. And we extracts parts of the examples into the documentaiton and we should not pollute those examples with things that are not really good "examples" on how you should add your DAGs. - and those example usually show the DAG definition/default_args as part of the example - by having the special decorator, or parameter on the DAG to indicate that DAG shoudl have "watcher" added migth be too easily copied from those extracted examples. Having explicit watcher makes it so much easier - it is at the and and it is explicit (which means there is no magic) But actually what made me think now - we should actually make it even more separated an explicit. @mnojek why don't we change the the watcher's local import and some comment there to make it even more separated and "explicit" for example: ```python >> delete_bucket ) from tests.system.utils.watcher import watcher # This test run as a system test needs watcher in order to mark success/failure # where "tearDown" task is part of the DAG list(dag.tasks) >> watcher() ``` ########## File path: scripts/ci/pre_commit/pre_commit_check_watcher_in_examples.py ########## @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import sys +from pathlib import Path +from typing import List + +from rich.console import Console + +if __name__ not in ("__main__", "__mp_main__"): + raise SystemExit( + "This file is intended to be executed as an executable program. You cannot use it as a module." + f"To run this script, run the ./{__file__} command [FILE] ..." + ) + + +console = Console(color_system="standard", width=200) + +errors: List[str] = [] + +WATCHER_APPEND_INSTRUCTION = "list(dag.tasks) >> watcher()" + +PYTEST_FUNCTION = """ +def test_run(): + from airflow.utils.state import State + + dag.clear(dag_run_state=State.NONE) + dag.run() +""" + + +def _check_file(file: Path): + content = file.read_text() + if "from tests.system.utils.watcher import watcher" in content: Review comment: Yeah. That check is "best effort" we could have import the AST and analyze it, but I think this is the 20/80 Pareto's rule in full swing. Even if we miss it once or twice, if somoene actually made an effort to import watcher they will likely not forget to add it as dependencies :) ########## File path: tests/system/providers/google/bigquery/example_bigquery_dataset.py ########## @@ -0,0 +1,94 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Example Airflow DAG for Google BigQuery service testing dataset operations. +""" +import os +from datetime import datetime + +from airflow import models +from airflow.operators.bash import BashOperator +from airflow.providers.google.cloud.operators.bigquery import ( + BigQueryCreateEmptyDatasetOperator, + BigQueryDeleteDatasetOperator, + BigQueryGetDatasetOperator, + BigQueryUpdateDatasetOperator, +) +from airflow.utils.trigger_rule import TriggerRule Review comment: Yeah - see the comments I added above. ########## File path: scripts/ci/pre_commit/pre_commit_check_watcher_in_examples.py ########## @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import sys +from pathlib import Path +from typing import List + +from rich.console import Console + +if __name__ not in ("__main__", "__mp_main__"): + raise SystemExit( + "This file is intended to be executed as an executable program. You cannot use it as a module." + f"To run this script, run the ./{__file__} command [FILE] ..." + ) + + +console = Console(color_system="standard", width=200) + +errors: List[str] = [] + +WATCHER_APPEND_INSTRUCTION = "list(dag.tasks) >> watcher()" + +PYTEST_FUNCTION = """ +def test_run(): Review comment: That's a nice idea. If pytest still discovers it this way, I lke it a lot better. also you can even connect it with local import to make it even more localized (same reason as in the other comment , where we care about "example_dag" being still an example dag despite also being a self-contained test. Also it nicely solves the problem which I thought about - what happens if we have two dags in an example. They would have to have slightiy different method ( dag1.clear() dag1.run() vs dag2.clear(() dag2.run()). Passing dag as parameter solves it nicely. I'd be for something like this: ``` # Needed to run the example dag as system test <link to the docs> from tests.airflow.proivders.util import get_test_run test_run = get_test_run(dag) ``` ########## File path: scripts/ci/pre_commit/pre_commit_check_watcher_in_examples.py ########## @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import sys +from pathlib import Path +from typing import List + +from rich.console import Console + +if __name__ not in ("__main__", "__mp_main__"): + raise SystemExit( + "This file is intended to be executed as an executable program. You cannot use it as a module." + f"To run this script, run the ./{__file__} command [FILE] ..." + ) + + +console = Console(color_system="standard", width=200) + +errors: List[str] = [] + +WATCHER_APPEND_INSTRUCTION = "list(dag.tasks) >> watcher()" + +PYTEST_FUNCTION = """ +def test_run(): Review comment: That's a nice idea. If pytest still discovers it this way, I lke it a lot better. also you can even connect it with local import to make it even more localized (same reason as in the other comment , where we care about "example_dag" being still an example dag despite also being a self-contained test. Also it nicely solves the problem which I thought about - what happens if we have two dags in an example. They would have to have slightiy different method ( dag1.clear() dag1.run() vs dag2.clear(() dag2.run()). Passing dag as parameter solves it nicely. I'd be for something like this: ```python # Needed to run the example dag as system test <link to the docs> from tests.airflow.proivders.util import get_test_run test_run = get_test_run(dag) ``` ########## File path: scripts/ci/pre_commit/pre_commit_check_watcher_in_examples.py ########## @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import sys +from pathlib import Path +from typing import List + +from rich.console import Console + +if __name__ not in ("__main__", "__mp_main__"): + raise SystemExit( + "This file is intended to be executed as an executable program. You cannot use it as a module." + f"To run this script, run the ./{__file__} command [FILE] ..." + ) + + +console = Console(color_system="standard", width=200) + +errors: List[str] = [] + +WATCHER_APPEND_INSTRUCTION = "list(dag.tasks) >> watcher()" + +PYTEST_FUNCTION = """ +def test_run(): Review comment: That's a nice idea. If pytest still discovers it this way, I lke it a lot better. also you can even connect it with local import to make it even more localized (same reason as in the other comment , where we care about "example_dag" being still an example dag despite also being a self-contained test. Also it nicely solves the problem which I thought about - what happens if we have two dags in an example. They would have to have slightiy different method ( dag1.clear() dag1.run() vs dag2.clear(() dag2.run()). Passing dag as parameter solves it nicely. I'd be for something like this: ```python # Needed to run the example dag as system test <link to the docs> from tests.airflow.proivders.util import get_test_run test_run = get_test_run(dag) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
