rjgoyln commented on PR #70620: URL: https://github.com/apache/airflow/pull/70620#issuecomment-5142059300
> Is it possible to further reduce the query with `UNION`? (SQLAlchemy `union_all`) I feel there’s a chance. Thanks for the suggestion. I benchmarked the `UNION ALL` approach. **Advantages** - Improves performance by about **35%** (~0.8 ms) on PostgreSQL 16. **Disadvantages** - Removes short-circuiting, requiring the database to compute XCom counts for all upstreams and Python to discard unfinished results afterward. - Makes the query and unpacking logic significantly more complex. I have the prototype ready. Let me know if you think the 0.8ms speedup is worth the added complexity, and I can push it! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
