GitHub user rahrajlat created a discussion: Display selected XCom values directly in DAG Graph View
**Problem** When monitoring a DAG run, the Graph View shows the state of each task, but sometimes additional runtime context is useful to understand what happened. For data pipelines, examples could include: * Rows processed * Files processed * Rows rejected * Data quality checks passed/failed * Output size * Number of models processed This information may already be available as XComs, but users need to open the task details, logs, or XCom view to see it. **For example:** <img width="1951" height="806" alt="image" src="https://github.com/user-attachments/assets/9ca88f5b-ee6f-40ce-8ccc-72a5e61c9fb2"; /> **Proposal** Provide an optional convention for a task to expose a small amount of runtime information directly on its node in Graph View. One possible implementation could use a reserved XCom key: context["ti"].xcom_push( key="__airflow_display", value={ "Rows": 8_421_932, "Rejected": 12_304, }, ) If this XCom is present for a task instance, Graph View could render those key/value pairs on the corresponding task node. If it is not present, the node behaves exactly as it does today. The API/key above is only an example. The main question is whether this capability would be useful and what the most Airflow-native implementation would be. **Why XCom?** The information is runtime-specific and belongs to a particular task instance and DAG run, which already aligns with the XCom model. Airflow would not need to know how these values are produced. For example: * A Glue/Spark task could expose records processed * A SQL task could expose rows affected * A data-quality task could expose passed/failed checks * A file-processing task could expose files processed * A dbt task could expose models/tests executed The task/operator would be responsible for producing the value. Airflow would only provide a standard mechanism for surfacing it in the Graph View. **Scope** I would expect this to remain intentionally small and optional: * Only simple scalar/string values * A small maximum number of displayed values * No automatic metric collection * No changes for tasks that do not opt in * XComs remain available normally through the existing UI The intention is to provide quick runtime context rather than turn Graph View into an observability dashboard. **Questions** 1. Would displaying a small amount of task-instance runtime information directly in Graph View be useful? 2. Would XCom be an appropriate mechanism, or is there an existing Airflow abstraction that would be a better fit? 3. Would a reserved XCom key be preferable, or should tasks explicitly mark existing XComs as displayable? 4. Are there concerns around Graph View performance or readability for large or dynamically mapped DAGs? If there is interest, I would be happy to build a small proof of concept to demonstrate the UX. GitHub link: https://github.com/apache/airflow/discussions/73691 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
