arifrhm commented on issue #76558: URL: https://github.com/apache/airflow/issues/76558#issuecomment-6669009407
Hi @zach-overflow, thanks for clarifying the context! I completely agree that these execution API and transaction boundary decisions deserve careful discussion. When looking at the two directions outlined in your issue: - **Option 1 (the PR):** Looked like a lightweight, low-risk fix/PoC that mirrors `_eager_load_dag_run_for_validation()` in the scheduler, addressing the immediate N+1 statement timeouts inside the `FOR UPDATE` lock without introducing breaking changes to the Task SDK wire protocol. - **Option 2 (deferring event loading / Task SDK lazy fetch):** Solves the broader issue (including the payload serialization cost of large `extra` fields), but involves deeper contract/protocol changes across Task SDK boundaries. I'm very happy to keep the PR as a concrete reference/PoC for Option 1 while we wait for feedback from maintainers (@ashb, @amoghrajesh) on which direction the community prefers. If the consensus leans towards Option 2 or a different architectural route, happy to pivot or follow the community's guidance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
