nathadfield opened a new issue, #60589: URL: https://github.com/apache/airflow/issues/60589
### Apache Airflow version 3.1.6 ### If "Other Airflow 3 version" selected, which one? _No response_ ### What happened? When using `BigQueryInsertJobOperator`, the Rendered Templates view only displays the full `configuration` field as JSON, without extracting the SQL query into a separate field. <img width="1684" height="413" alt="Image" src="https://github.com/user-attachments/assets/7209f583-df8c-46dd-9b58-18969932156c"; /> The SQL query is embedded within the JSON configuration as an escaped string (e.g., `"query": "\\nSELECT\\n customer_id,\\n..."`), making it impossible to: - Quickly copy the SQL query for debugging or testing - View the SQL with proper formatting and syntax highlighting - Avoid dealing with JSON escape characters This is a regression from the expected behaviour where nested template fields defined in `template_fields_renderers` should be displayed as separate entries. ### What you think should happen instead? The Rendered Templates view should display **two separate entries**: 1. **`configuration`** - The full job configuration as formatted JSON 2. **`configuration.query.query`** - The extracted SQL query with SQL syntax highlighting This would match the operator's `template_fields_renderers` configuration: ```python template_fields_renderers = { "configuration": "json", "configuration.query.query": "sql" # Should create a separate SQL entry } ``` Users should be able to: - See the SQL query as a standalone field with proper SQL formatting - Copy the entire SQL with one click using the copy button - Avoid manually extracting SQL from nested JSON strings ### How to reproduce 1. **Create a DAG** with `BigQueryInsertJobOperator` containing a multi-line SQL query: ```python from datetime import datetime from airflow import DAG from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator COMPLEX_SQL = """ SELECT customer_id, customer_name, order_date, SUM(order_amount) as total_amount, COUNT(DISTINCT order_id) as order_count FROM `project.dataset.orders` o INNER JOIN `project.dataset.customers` c ON o.customer_id = c.id WHERE order_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) AND order_status = 'completed' GROUP BY customer_id, customer_name, order_date ORDER BY total_amount DESC LIMIT 100 """ with DAG( dag_id="bigquery_sql_rendering_test", start_date=datetime(2024, 1, 1), schedule=None, catchup=False, ) as dag: run_query = BigQueryInsertJobOperator( task_id="run_query", configuration={ "query": { "query": COMPLEX_SQL, "useLegacySql": False, } }, project_id="my-project", location="US", ) ``` 2. **Trigger the DAG** manually from the Airflow UI 3. **Wait for the task to complete** (it will likely fail due to invalid credentials, but that's okay) 4. **Navigate to**: Grid View → Click on the task instance → "Rendered Templates" tab 5. **Observe**: Only the `configuration` field is shown as JSON with the SQL embedded inside **Expected**: Should see both `configuration` (JSON) and `configuration.query.query` (SQL) as separate entries **Actual**: Only see `configuration` (JSON) with SQL embedded as an escaped string within ### Operating System Debian GNU/Linux 12 (bookworm) ### Versions of Apache Airflow Providers _No response_ ### Deployment Other ### Deployment details _No response_ ### Anything else? This issue is related to: - [#39527](https://github.com/apache/airflow/issues/39527) - Restored the Rendered Templates view in Grid - [#49064](https://github.com/apache/airflow/issues/49064) - Added `template_fields_renderers` support in UI While both issues were resolved, the specific problem of extracting nested template fields (using dot notation like `configuration.query.query`) remains unaddressed. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
