potiuk commented on code in PR #34006: URL: https://github.com/apache/airflow/pull/34006#discussion_r1313034507
########## docs/apache-airflow/best-practices.rst: ########## @@ -176,6 +176,96 @@ Good example: In the Bad example, NumPy is imported each time the DAG file is parsed, which will result in suboptimal performance in the DAG file processing. In the Good example, NumPy is only imported when the task is running. +Since it is not always obvious, see the next chapter to check how my code is "top-level" code. + +How to check if my code is "top-level" code +------------------------------------------- + +In order to understand whether your code is "top-level" or not you need to understand a lot of +intricacies of how parsing Python works. In general, when Python parses the python file it executes +the code it sees, except (in general) internal code of the methods that it does not execute. + +It has a number of special cases that are not obvious - for example top-level code also means +any code that is used to determine default values of methods. + +However, there is an easy way to check whether your code is "top-level" or not. You simply need to +parse your code and see if the piece of code gets executed. + +Imagine this code: + +.. code-block:: python + + from airflow import DAG + from airflow.operators.python import PythonOperator + import pendulum + + + def get_task_id(): + return "print_array_task" # <- is that code going to be executed? + + + def get_array(): + return [1, 2, 3] # <- is that code going to be executed? + + + with DAG( + dag_id="example_python_operator", + schedule=None, + start_date=pendulum.datetime(2021, 1, 1, tz="UTC"), + catchup=False, + tags=["example"], + ) as dag: + + operator = PythonOperator( + task_id=get_task_id(), + python_callable=get_array, + dag=dag, Review Comment: Could be .. but the mechanism of checking it is the same. If they add `print` in my_func() they will also see it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
