Hello, As we implement deferrable capabilities for operators, we're going to want to update the sample dags/system tests. The problem we encounter at the moment is that system tests are run with pytest using the DebugExecutor, and it only runs the "worker part" of the operators, without handling the "deferred part" that is taking place in the Triggers. In system tests, a task ran in deferrable mode just hangs forever in the “deferred” state, never completing.
I see 3 approaches to that problem: 1. Patch the DebugExecutor (or an other executor that'd become the "official" executor for system tests) to handle deferred tasks. This bears some complexity, but is probably doable with some ad-hoc code that won't be the same as the "production" code but could execute the right things in order (Trigger.run after the TaskDeferred exception, and then the "execute_complete" method with the event returned) 2. Change the way we run system tests, stop using pytest, and start airflow in breeze, scheduling the example dags as actual dags. This has the upside of being the closest to prod, but it's also a more resource-intensive solution, and gathering the success/failure status and the logs will require some piping. 3. Do nothing, and forbid the use of deferrable in system tests. This is not great for obvious reasons. It's a bit hard to say which option is "the best" before digging into it, but my preference would go to 1. from what I tell so far. What do you community think about those options ? Did I forget a potential solution ? Other providers who also have system tests and deferrable operators, what is your approach ?
