josh-fell commented on issue #16529: URL: https://github.com/apache/airflow/issues/16529#issuecomment-864280192
Hi @gfelot, Addressing the errors you are observing: - The documentation you are referring to is for version 2.1 (current stable) while you are running 2.0.2 (of which the documentation can be found [here](https://airflow.apache.org/docs/apache-airflow/2.0.1/_api/airflow/models/dag/index.html?highlight=dag#module-airflow.models.dag)). The `render_template_as_native_obj` parameter wasn't introduced until 2.1 so that's why it's missing from the `DAG` object within your environment. Additionally, the `jinja_environment_kwargs` arguments refer to the initialization parameters for a [jinja2.Environment](https://jinja.palletsprojects.com/en/3.0.x/api/#jinja2.Environment). - The [documentation for 2.1](https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/models/dag.html#DAG) _is_ missing for the `render_template_as_native_obj` parameter and we'll get that added. Thank you for catching this! Pertaining to splitting Jinja strings: - Jinja expressions are not evaluated/rendered in Airflow until runtime (i.e. within the operator's `execute()` method. As written, the parsing/splitting of the string is done outside of the operator. Only after the task begins executing is the Jinja expression evaluated and rendered. Therefore the value being initialized is `[" {{ dag_run.conf['file_path'] }} "]` as there is no `/` character in the literal string of `{{ dag_run.conf['file_path'] }}`. - A couple options: - Assuming the "test_var" task is not for demonstration purposes, move the splitting of the `file_path` value into the `print_var()` function and use `XComs` or the Taskflow API to pass values to the "file_verification_fail_to_s3" task. - Since the `source_bucket_key`, `source_bucket_name`, and `dest_bucket_name` arguments can be templated values in the `S3CopyObjectOperator`, you could directly access the `conf` object as you do now and split the value as needed within the Jinja expression. For similar questions in the future, please use GitHub discussions, not issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
